Skip to main content

# Questions tagged [linear-regression]

For questions about linear regressions, an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables.

55 questions
Filter by
Sorted by
Tagged with
20 votes
1 answer
36k views

### what is the variance of a constant matrix times a random vector?

$\newcommand{\Var}{\operatorname{Var}}$In this video is claimed that if the equation of errors in OLS is given by: $$u=y - X\beta$$ Then in the presence of heteroscedasticity the variance of $u$, will ...
• 323
5 votes
2 answers
842 views

• 483
0 votes
2 answers
2k views

### Proof $E[\hat \sigma ^2] = E\left( \frac{1}{n-2} \Sigma(y_i-\hat{y_i})^2 \right) = \sigma ^2$: Linear Regression

I am trying to prove that the estimated variance of the residual $$\hat \sigma ^2 = \frac{\Sigma(y_i-\hat{y_i})^2}{n-2}$$ is an unbiased estimator of the variance of the error $\sigma^2$. So far ...
• 5,175
7 votes
4 answers
3k views

### Why does $A^TAx = A^Tb$ have infinitely many solution algebraically when $A$ has dependent columns?

This is a problem from least square approximation, where we solve the equation $A^TAx = A^Tb$ when $Ax = b$ is unsolvable. The case I am dealing with is when A has dependent columns, i.e. A is an m by ...
• 181
6 votes
1 answer
2k views

### Connection Between Orthogonal Projection onto the Unit Simplex and the Softmax Function

Referring to papers Softmax to Sparsemax and Efficient Projections onto the L1-Ball, what is the relationship between a euclidean projection onto the probability simplex and applying the Softmax ...
• 91
3 votes
1 answer
4k views

• 333
0 votes
2 answers
158 views

### Differentiate matrix expression (linear regression)

$$\frac{d}{dw} [w^TX^TXw - 2w^TX^Ty+y^Ty] = 2(X^TXw-X^Ty)$$ I do not understand how the RHS was obtained -- are there certain matrix differentiation properties which can be used to show this? Why does ...
0 votes
2 answers
9k views

• 213
2 votes
2 answers
3k views

### Proof that sum of squares of error for simple linear regression follows chi-square distribution

I can understand that if Y1~Yn are random samples from N(μ,σ), then the sum of squares of difference between Yi and bar(Y) divided by sigma^2 follows chi-square distribution with n-1 degress of ...
• 141
2 votes
1 answer
156 views

### For a linear regression of $\{(i,y_i)\}_{i=0}^{n-1}$, where $(y_i)$ is increasing and non-negative, is the $y$-intercept at least $-y_{n-1}$?

Suppose we have a set of data points $\{(i,y_i)\}_{i=0}^{n-1}$, where $y_i$ are non-negative integers and where $(y_i)_{i=0}^{n-1}$ is an increasing sequence. Question: In a simple linear regression ...
1 vote
2 answers
286 views

### Matrix Derivative of Tichonov Regularization Operator

I'm not very familiar with matrix derivative and was wondering what are the first two derivatives of the map $$X\mapsto (X^TX + \lambda I)^{-1}X^Ty,$$ should be; where $y$ is a fixed vector and $X$ ...
• 6,808
1 vote
1 answer
57 views

• 405
1 vote
1 answer
869 views

0 votes
0 answers
95 views

### Instrumental Variable Regression Questions

We're learning about instrumental variable estimators and I just want to confirm my logic in these answers. Really appreciate the help! a. True- If this happens and these variables are omitted, it ...
0 votes
1 answer
2k views

### Convert log model to linear model

I have a log-log model as follows: ln quality = ln price + predictor_2 + predictor_3 I ran a regression and using the coefficient values obtained, I predicted log quality values and then I plotted ...
• 101
0 votes
1 answer
930 views

### Proving $a$ (in $Y = aX + b + e$) satisfies $a = Cov(X, Y )/Var(X)$

In a linear regression model, we postulate that random variables $X$ and $Y$ are related by $$Y = aX + b + e$$ where a and b are constants (called the regression coefficients) and e (representing ...
• 11
0 votes
1 answer
341 views

### ANOVA - Distribution of $\hat{\beta}_1$ still exists although $\beta_1=0$ under $H_0$?

We're doing simple linear regression. Anova decomposition. $$SS=RegSS+RSS$$ or better $$(n-1)s_Y^2=\hat{b}_1^2s_{XX}+(n-2)s^2$$ We know that when the fit is good, then RegSS will be large. Therefore ...
• 5,267
0 votes
1 answer
205 views

### Fitting a line to the set of planes

The methods of fitting lines, planes to the set of points are rather popular. But is it possible to do anything similar for the case when the 3D line is fitted to the set of 3D planes? I.e. there are ...
0 votes
2 answers
615 views

### Matrix Least Squares on the Rows Instead of Columns

I want to solve the equation $AB=C, A\epsilon \mathbb{R}^{3\times 3}, B\epsilon \mathbb{Z}^{3\times 10}, C\epsilon \mathbb{R}^{3\times 10}$ for $A$. The solution should minimize the norm (best is ...
• 99
0 votes
1 answer
59 views

• 1,381