Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.

learn more… | top users | synonyms

0
votes
0answers
2 views

Softmax Regression Gradient Derivation

I'm implementing softmax regression and am deriving the max-log-likelihood update for gradient descent by hand first. Coming from the Stanford UFLDL site, they show the gradient of the cost function ...
1
vote
2answers
51 views

properties of least square estimators in regression

$Y_i=\beta_0+\beta_1 X_i+\epsilon_i$ where $\epsilon_i$ is normally distributed with mean $0$ and variance $\sigma^2$ . The least square estimators of this model are $\hat\beta_0$ and $\hat\beta_1$. ...
1
vote
0answers
14 views

LRT and Wald Test in Multivariate Linear Regression significance

I've been researching all afternoon trying to get a better idea of what the Likelihood ratio test and the Wald test are actually doing. I have a bunch of covariates and I'm testing out like 30 ...
0
votes
0answers
16 views

Finding the curvature from a set of datapoints

I have a set of 1. 1-d 2. 2-d data. I want to find the curvature at each single point. Till now I was using difference technique to find out the curvature, i.e, central difference at middle and ...
1
vote
1answer
19 views

Significance of dummy variables in probit regression

I am an undergraduate student working on some projects using probit regression. I have a question on dummy variables that I was hoping someone could help me with (which I think stems from an ...
1
vote
1answer
14 views

Find a distribution for this plot

Please help me find a formula that fits the distribution. It does not need to be exact, a simple approximation would suffice. Bonus points if you can tell me which predefined distribution in the ...
0
votes
2answers
152 views

Non-linear regression - least square regression

I was trying to get some insight into how to solve non-linear regression model problems. Unfortunately, I've never attended a lecture on statistical math. Here is the link: In page number 4, they ...
0
votes
1answer
21 views

finding a question about constrained regression - had a side constraint $x \geq y^*z$

I saw a question that asked about solving a non-negative least squares problem with $3$ unknowns, $(x,y,z)$. But there was an additional constraint, $x \geq y^*z$. Would appreciate getting the ...
0
votes
1answer
23 views

Getting VAR parameters from a research paper.

Many econometrics papers provide the parameters used in their VAR model. If I notate my VAR model as $$z_{t+1} = c + B z_{t} + \Sigma \epsilon_{t+1}$$ where $\epsilon \sim N(0, I)$, then I need to ...
0
votes
2answers
74 views

Non-linear regression fit

I'm trying to fit my data to the following equation: $$ Y = A(1-2e^{bx}) $$ What I tried to do was transform the equation to a linear form via the following steps: \begin{align*} & A-Y = ...
0
votes
1answer
28 views

Least squares with known error in y

so I want to do a linear least squares regression on my data, however I have known experimental error on my data points in $y$ and relatively few numbers of points so I would like to use values ...
0
votes
1answer
29 views

How do I solve for vector $P$ in the matrix equation $s=A'B^{-1}A$?

I would like to rearrange the matrix equation $s=A'B^{-1}A$ into the form $A=f(s,B)$ (i.e., some function of $s$ and $B$), where s is scalar, $A$ is $n\times 1$, $A'$ is the transpose of $A$, and $B$ ...
1
vote
1answer
144 views

Optimizing Independent Variables to Maximize Dependent Variable

I looked around online and couldn't find anything that was answering my question so I thought I would take to the stack! I'm interested in knowing if there is a statistical or mathematical way of ...
1
vote
3answers
47 views

How come least square can have many solutions?

I know there always exists a least-square solution $\hat{x}$, regardless of the properties of the matrix $A$. However, I keep finding online that least-square can have infinitely many solutions, if ...
0
votes
1answer
8 views

In a simple regression model estimated using OLS, the covariance between the estimated errors and regressors is zero by construction

Is this statement true or false? I seem to remember that this relationship does not hold when the regression has no intercept, however my teacher said that this was true regardless of whether we ...
2
votes
1answer
366 views

Understanding Regularization parameters in Machine Learning/Statistics

Suppose I have the following $k$ degree polynomial regression model with a data set of size $n$ which includes a $k$-dimensional feature vector $x$ and an outcome denoted $t_i$ for each vector in the ...
-1
votes
0answers
19 views

(x/y) v (y/x) as predictor in regression

I am trying to predict a variable 'score' with x and y. I believe it is related to (x/y)/z and z/(x/y), but I'm not sure which. Here is some data: ...
0
votes
1answer
25 views

Polynomial least squares fit — restrictions on order?

If we're finding an interpolating polynomial for 10 data pairs, the order of the polynomial has to be 9. In class, my professor said that when doing a polynomial least squares fit, if you have 10 ...
1
vote
2answers
62 views

Fit exponential with constant

I have data whic would fit to an exponential function with a constant. So y=aexp(bt) + c Now I can solve an exponential without a constant using least square by taking log of y and making the ...
3
votes
0answers
26 views

How to perform nonlinear regression with regressors affected by gaussian error?

I am trying to calibrate a sensor and I have a data set consisting of several observations of a 3-dimensional vector $X_i$, with $X_i=w_i + \epsilon_i$ where $w_i$ is the value that the sensor ...
0
votes
1answer
11 views

Least Squares Estimators Derivation (Bi-variate)

To derive least square estimators: We have $SS(\alpha,\beta)= \sum(y_i-\alpha-bx_i)^2$ and find partials for each. The answer I get is: $\beta = \frac{\sum y_i-\bar{y}}{\sum x_i-\bar{x}}$, but the ...
1
vote
1answer
55 views

Why do the components of an equivalent kernel sum to 1?

Let $\textbf{x} = (x_1, \dots, x_n)^T \in \mathbb{R}^n$ and $k \in \mathbb{N}$. We define $$ X := \begin{pmatrix} 1 & x_1 & \cdots & x_1^k \\ \vdots & \vdots & & ...
2
votes
1answer
55 views

Outlier detection with robust multiple regression model

I have a set of features (eg, location, income, budget, education) that I use to predict a continuous variable (say, amount spent per day on the internet). I am interested in detecting outliers. I ...
1
vote
1answer
26 views

Multiple Linear Regression in Matrix Form

I am currently studying for my exams and came across the following question: State the multiple linear regression equation in matrix form. Write down the order of each matrix and explain what the ...
0
votes
2answers
24 views

Calculating the correlation coefficient between least square estimates

PROBLEM STATEMENT: Consider the following 2-variable linear regression where the error $e_i$ 's are independently and identically distributed with mean $0$ and variance $1$; $$y_i = α + β(x_i − \bar ...
1
vote
1answer
24 views

I would like to know how to do log transformation of hyperparameters in Gaussian Process Classification.

I am using Gaussian Process classification and I want to do log transform of the hyperparameters so that they are all positive. From this www.lce.hut.fi/research/mm/gpstuff/GPstuffDoc.pdf document, I ...
2
votes
0answers
20 views

Can I adjust linear growth of a a subpopulation to a linear decay of the general population?

I need to estimate the amount of CF patients in Poland in the next four years. I have: estimations of the Polish population for the future years a CF patients' register for the last couple of years ...
-1
votes
0answers
15 views

Sample variance estimator OLS without intercept

In the classical OLS regression model the estimator of the variance is $s_n^2=\frac{1}{n-k}\sum_{i=1}^n \hat{e}_i$ where $k$ is the number of regressors without the constant. What happens to this ...
0
votes
1answer
24 views

Why is the SSE <= SST?

I can think of a regression line producing a larger sum of squared errors (SSE) than the total sum of squares (SST). I read that should not be possible, how come? My understanding is that the ...
0
votes
0answers
20 views

Transpose is just the way of generalizing a dot product?

It seems like $a^Tb$ is the same as writing $a \cdot b$ in matrix form. 1) Why is $n \times 1$ and $n \times 1$ matrix multiplication undefined? 2) Is this just a generalization of the dot ...
1
vote
1answer
12 views

Do we only use T distribution for confidence interval of Beta for linear regression or we can also normal distribution?

Do we only use T distribution for confidence interval of Beta for linear regression or we can also normal distribution? Is it that when sample size is less than 30 then we use T distribution else ...
4
votes
5answers
42 views

calculating least squares fit

I read this thread talking about 'why we use least squares' for curve fitting Why do we use a Least Squares fit? One answer by Chris Taylor begins with the assumption that we should look for $$ ...
1
vote
4answers
49 views

How to estimate $\alpha$ in $y=(1-\exp(-\alpha x))/(1+\exp(-\alpha x))$?

I have a function $$y=\dfrac{1-\exp(-\alpha x)}{1+\exp(-\alpha x)}$$ where $y$ is not binary. The range of this function is $[-1,1)$. So this does not fit into either logit or probit models. How ...
0
votes
1answer
42 views

Identical observations in linear regression

I want to do a linear regression $Y = X\beta + e$, but some of the observations (rows in $X$) are identical (about 30 000 out of 50 000 remain after deleting all duplicates), so when I try to ...
0
votes
1answer
17 views

Method for ?not quite? weighted least squares fitting for more realistic results

I need a linear least squares type of fitting algorithm that understands how to weight the probability of a response coming from certain functions over another. To explain, given the standard linear ...
1
vote
1answer
28 views

Best percentage change for trend

Consider the revenue of a company for the last five year and you want to to know whether there is an upward, downward or no trend. How would you calculate the "optimal" percentage change? I have an ...
1
vote
1answer
15 views

Assumption of Normal Distribution

I have a problem and I do not know when it is crucial and when it is NOT crucial to assume a normal distribution regarding linear regression, for estimates, t-tests, f-tests, confidence intervals and ...
-2
votes
0answers
21 views

Repeated measurements for multiple groups with multiple samples

I have 4 groups with 5-6 samples in each, and I have repeated a measurement on them 6 times over a time. This is done during an aging test and the results are evolving over the time roughly linearly. ...
0
votes
2answers
344 views

Extrapolation with exponential curve

I would like to extrapolate time series using exponential curve while getting the parameters via linear regression. Exponential curve is given as $g=e^{~a + b \cdot t}$. Since I want to use linear ...
1
vote
1answer
28 views

Is there such a thing as a weighted multiple regression?

I'm new to linear algebra, but I know how multiple linear regressions work. What I want to do is something slightly different. As an example, let's say that I have a list of nutrients I want to get ...
1
vote
0answers
34 views

Confusion with Bayesian Linear Regression

In the book Gaussian Processes for Machine Learning in Chapter 2 p. 11 (see http://www.gaussianprocess.org/gpml/chapters/RW2.pdf), eq. 2.9 states: $p(f_* | X, y) = \int p(f_* | x_*,w) p(w|X, y)dw$ ...
2
votes
1answer
38 views

Calculate trend and represent in text?

First off, I'm terrible at math. I'm writing a script that monitors transactions from clients daily over a 7 day period. Given a set of numbers like below, I would like to calculate a trend and ...
0
votes
0answers
21 views

Relation between the Coefficient of Multiple Correlation and Coefficient of Simple Correlation

Consider the regression model $Y=\beta_1 X_1+\beta_2 X_2+\epsilon$, with a sample of size $n$, $Y_i=\beta_1 X_{i1}+\beta_2 X_{i2}+\epsilon_i$, $\epsilon_i \in N(0,\sigma^2)$. Suppossing ...
0
votes
1answer
35 views

Multiple linear regression inconsistency?

I've got a linear model: $y_i=β_1x_{i1}+β_2x_{i2}+ε_i$ where E($ε_i$)=0 and Var($ε_i$)= $σ^2I_n$ for i=1,...,n Supposed we don't have the data for $x_{i2}$ and we estimate: $y_i=β_1x_{i1}+ε_i$ for ...
1
vote
1answer
21 views

Logistic regression coefficients problem

I'm using logistic regression model to do a multi-class classification (4 classes). I want to look at the logistic regression coefficients to see the importance of different features. I got model ...
0
votes
1answer
28 views

Distribution of e, least squares residuals

I have the model $y=X{\beta}+{\epsilon}$ and $E({\epsilon})=0$ and $Var({\epsilon})={\sigma}^2I_n$ The vector y can be written: $y=X\hat{\beta}+{e}$ If ${\epsilon}$~$N(0, {\sigma}^2I_n)$ how is ...
0
votes
1answer
26 views

Quadratic Form Matrices

How do I know if a matrix in quadratic form, e.g. D'MD is positive or negative (semi)definite? M here is the residual maker matrix for X, so I know that it is symmetric. I know what the definitions ...
2
votes
0answers
27 views

Show $\hat{\beta}$ and $s^2$ are independent?

I have the model: $y=X{\beta}+{\epsilon}$ I know $\hat{\beta}=(X'X)^{-1}X'y$ and that it is an unbiased estimator of ${\beta}$ and that $s^2=\hat{\epsilon}'\hat{\epsilon}/(n-k)$ and is an unbiased ...
1
vote
1answer
369 views

Derivative of logistic loss function

I am using logistic in classification task. The task equivalents with find $\omega, b$ to minimize loss function: That means we will take derivative of L with respect to $\omega$ and $b$ (assume y ...
2
votes
1answer
311 views

Multiple linear regression with interaction

I'm doing a multiple linear regression with interacting variables. I'll give you an example: $y$=value, $x_1$=material, $x_2$=weight, $x_3$=color $x_1$ and $x_2$ are interacting variables but $x_3$ ...