Questions tagged [linear-regression]
For questions about linear regressions, an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables.
55
questions
20
votes
1
answer
36k
views
what is the variance of a constant matrix times a random vector?
$\newcommand{\Var}{\operatorname{Var}}$In this video is claimed that if the equation of errors in OLS is given by:
$$u=y - X\beta$$
Then in the presence of heteroscedasticity the variance of $u$, will ...
- 323
5
votes
2
answers
842
views
Prove uniqueness of solutions of different OLS matrix cases
Let $D = \{(x_1, y_2), (x_2, y_2), \ldots , (x_n, y_n)\}$ where $x_i \in \mathbb{R}^d$ and $y_i \in \mathbb{R}$. One may use linear regression to predict $y$ as $w^Tx$ for some parameter vector $w \in ...
- 105
18
votes
3
answers
32k
views
Proof that trace of 'hat' matrix in linear regression is rank of X
I understand that the trace of the projection matrix (also known as the "hat" matrix) X*Inv(X'X)*X' in linear regression is equal to the rank of X. How can we prove that from first principles, i.e. ...
- 283
5
votes
1
answer
1k
views
Basic exponential regression
Background:
I'm attempting to learn about basic statistics for the infrastructure asset management industry:
Basic math explanation (related to estimating linear regression with no intercept)
...
- 425
1
vote
1
answer
3k
views
Leverage in simple linear regression
I am trying to understand how to calculate the leverage in a simple linear regression (just $1$ independent variable).
The leverage at value $X=x_i$ is known to be $h_{ii}=\frac{1}{n}+\frac{(x_i-\...
- 483
0
votes
2
answers
2k
views
Proof $E[\hat \sigma ^2] = E\left( \frac{1}{n-2} \Sigma(y_i-\hat{y_i})^2 \right) = \sigma ^2$: Linear Regression
I am trying to prove that the estimated variance of the residual
$$\hat \sigma ^2 = \frac{\Sigma(y_i-\hat{y_i})^2}{n-2}$$
is an unbiased estimator of the variance of the error $\sigma^2$.
So far ...
- 5,175
7
votes
4
answers
3k
views
Why does $A^TAx = A^Tb$ have infinitely many solution algebraically when $A$ has dependent columns?
This is a problem from least square approximation, where we solve the equation $A^TAx = A^Tb$ when $Ax = b$ is unsolvable.
The case I am dealing with is when A has dependent columns, i.e. A is an m by ...
- 181
6
votes
1
answer
2k
views
Connection Between Orthogonal Projection onto the Unit Simplex and the Softmax Function
Referring to papers Softmax to Sparsemax and Efficient Projections onto the L1-Ball, what is the relationship between a euclidean projection onto the probability simplex and applying the Softmax ...
- 91
3
votes
1
answer
4k
views
derivative transpose
I'm reading the book "The Elements of Statistical Learning - Data Mining, Inference, and Prediction" chapter 3 and there comes a simple derivation that I don't understand:
We have: $...
0
votes
1
answer
2k
views
For the simple linear regression model, show that the elements of the hat matrix $H$ are...
Need some help with this one.
For the simple linear regression model, show that the elements of the hat matrix $H$ are:
$h_{ij}=1/n + (x_i -\bar x)(x_j -\bar x)/S_{xx}$ and
$h_{ii}=1/n + (x_i -\bar x)^...
- 333
0
votes
2
answers
158
views
Differentiate matrix expression (linear regression)
$$\frac{d}{dw} [w^TX^TXw - 2w^TX^Ty+y^Ty] = 2(X^TXw-X^Ty)$$
I do not understand how the RHS was obtained -- are there certain matrix differentiation properties which can be used to show this? Why does ...
- 1,563
0
votes
2
answers
9k
views
How to calculate the variance of the error term in least squares method for simple linear regression?
We have
$$y_i = \beta_0 + \beta_1x_i + \epsilon_i$$ and $$ \hat{y_i} = \hat{\beta_0} + \hat{\beta_1}x_i $$
where $\epsilon_i \sim N(0, \sigma^2)$.
Let $$e_i = y_i - \hat{y_i} $$
I showed that
$$E(...
- 2,543
14
votes
2
answers
10k
views
why is the least square cost function for linear regression convex
I was looking at Andrew Ng's machine learning course and for linear regression he defined a hypothesis function to be $h(x) = \theta_0 + \theta_1x_1 + \dots + \theta_nx_n$, where $x$ is a vector of ...
- 421
7
votes
1
answer
5k
views
Explain about the Correlation of Error Terms in Linear Regression Models
I would like to ask for the interpretation, both mathematically and intuitively if possible, about the homoscedasticity of the variance of errors in linear regression models.
If there is correlation ...
- 405
6
votes
0
answers
153
views
Prime number intercept
Suppose I arrange my (infinite) list of prime numbers in the following way: \begin{array}{c|c}x_i&2&5&11&17&23&31&\cdots\\\hline y_i&3&7&13&19&29&37&...
- 27.1k
5
votes
2
answers
1k
views
How can I get the gradient of the normal equation for weighted linear regression?
The normal equation for weighted linear regression looks like this:
$$J(\theta) = (X\theta - y)^TW(X\theta - y),$$ where $X\in\Re^{m\times n}$, $\theta\in\Re^{n\times n}$, $y\in\Re^{m\times 1}$, $W\...
- 209
4
votes
1
answer
2k
views
Variance Estimate in linear regression
In a linear regression, $y=X\beta+\epsilon$, where $\epsilon\sim N(0, \sigma^2)$, $X\sim R^{N \times (p+1)}$. Assume the observations $y_i$ are uncorrelated and have constant variance $\sigma^2$, and ...
- 51
4
votes
1
answer
6k
views
If $Y=X\beta+\epsilon$, prove that the least square estimator $\hat\beta$ is independent of $Y-X\hat{\beta}$
Let $Y=X\beta+\epsilon$, where $Y$ is an $n$ by $1$ vector, $X$ is an $n$ by $p$ matrix with full rank and $\epsilon$ is an $n$ by 1 vector of random errors independently and normally distribution ...
- 1,306
3
votes
0
answers
1k
views
Proving that $\mathbf{(H-\frac{1}{n}J_n)}$ is indempotent
I am trying to show that the matrix $\mathbf{(H-\frac{1}{n}J_n)}$ is idempotent where $\mathbf{H}$ is the Hat-matrix (Projection matrix) of linear regression and $J_n$ is the $n\times n$ matrix with $...
- 21.4k
3
votes
1
answer
5k
views
How to prove sum of errors follow a chi square with $n-2$ degree of freedom in simple linear regression
In simple linear regression, the model is
\begin{equation}
Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i
\end{equation}
where $\varepsilon_i$ are i.i.d., and
\begin{equation}
\varepsilon_i \sim N(0, \...
- 133
3
votes
1
answer
2k
views
$\hat{Y} = X^T\hat{\beta}$ Matrix Dimension For Linear Regression Coefficients $\beta$
While reading about least squares implementation for machine learning I came across this passage in the following two photos:
Perhaps I’m misinterpreting the meaning of $ \beta $ but if $ X^T$ has ...
- 839
3
votes
2
answers
157
views
Basic math explanation (related to estimating linear regression with no intercept)
I have a question on the Cross Validated Stack Exchange site where I ask how to update the exponential regression coefficient of a vertically translated depreciation curve.
A Cross Validated community ...
- 425
2
votes
1
answer
461
views
how fit a model with data following asymptotic / sigmoid pattern
I'm trying to fit data. I assume that the association between dependent and indepdent variable is of the form
$$T(y)=aR(x)+b$$
I also know that my data are ressemble either an asymptotic function ...
- 1,005
2
votes
2
answers
385
views
Robust Least Squares for general 2D lines
Question
My goal is to robustly estimate a general 2D line from $n$ data points, where the line is parameterized by $\rho > 0$, the distance from the origin to the line and $\varphi$, the angle ...
- 283
2
votes
1
answer
263
views
How to fit a cumulative time series?
Assume we have a time series of $N$ different points $(t_i,y_i)$, for $i \in \{1,..N\}$. The number $N$ is "small" (let's say $3<N<10$, just to give an idea), so you may want to make ...
- 2,123
2
votes
2
answers
483
views
Optimality of the MSE in gaussian linear regression
Let's call $\hat{\beta}$ the least squares estimator of $\beta$ in the regression problem $Y = X\beta + \epsilon$ where $\epsilon \sim \mathcal{N}(0, \sigma^2)$.
In a statistics course, I get this ...
user485036
2
votes
2
answers
2k
views
Linear Regression: linear or reciprocal function?
The problem is given below:
Simultaneous values of time $t$ and output $y$ from a specific sensor has been measured and is tabulated below $$\begin{array}{cc}
t & y \\
\hline
1 & 17 \\
2 ...
- 213
2
votes
2
answers
3k
views
Proof that sum of squares of error for simple linear regression follows chi-square distribution
I can understand that if Y1~Yn are random samples from N(μ,σ), then the sum of squares of difference between Yi and bar(Y) divided by sigma^2 follows chi-square distribution with n-1 degress of ...
- 141
2
votes
1
answer
156
views
For a linear regression of $\{(i,y_i)\}_{i=0}^{n-1}$, where $(y_i)$ is increasing and non-negative, is the $y$-intercept at least $-y_{n-1}$?
Suppose we have a set of data points $\{(i,y_i)\}_{i=0}^{n-1}$, where $y_i$ are non-negative integers and where $(y_i)_{i=0}^{n-1}$ is an increasing sequence.
Question: In a simple linear regression ...
1
vote
2
answers
286
views
Matrix Derivative of Tichonov Regularization Operator
I'm not very familiar with matrix derivative and was wondering what are the first two derivatives of the map
$$
X\mapsto (X^TX + \lambda I)^{-1}X^Ty,
$$
should be; where $y$ is a fixed vector and $X$ ...
- 6,808
1
vote
1
answer
57
views
W and b for LMMSE using covariance(XY)
I would like to calculate the $W_{LMMSE}$ and $b_{LMMSE}$ for X which is a uniform random variable between $-\pi/2$ and $\pi/2$ and $Y=\sin(X)$.
I have the following info:
$\Sigma_{XY} = 2/\pi$
$\...
- 331
1
vote
1
answer
106
views
How the regression method affects the smoothness of the ’regression curve’ when we used them as a smoothing method?
The following Quizzes are the rough translation (with minor modification) of Quizzes No.08 of the exam of the "2019's semi-first grade of Japan Statistical Society Certificate (JSSC)" (See (ref 1) ). ...
- 665
1
vote
2
answers
6k
views
Proving that SSE and SSR are independent [duplicate]
I'm trying to show that SSE and SSR are independent (conditionally on X) but I have to use the following steps/hint.
[Hint: Notice you have to consider SSE and SSR as random variables, so be careful ...
- 135
1
vote
1
answer
563
views
Bayesian Regression Simplifying the posterior
In a linear regression model ${\bf y} = {\bf x}^T {\bf w} + \epsilon$, assuming Gaussian noise ($\epsilon\sim N(0,\sigma_n^2)$), and Gaussian priors on the weights (${\bf w}\sim N({\bf 0}, \Sigma_p))$,...
- 173
1
vote
1
answer
718
views
Standard Error of Coefficients in simple Linear Regression
In the book "Introduction to Statistical Learning" page 66, there are formulas of the standard errors of the coefficient estimates $\hat{\beta}_0$ and $\hat{\beta}_1$. I know the proof of $SE(\hat{\...
- 405
1
vote
1
answer
869
views
Finding a solution with minimal $\ell_2$ norm in linear regression with dependent variables
I am trying to find a linear regression for the problem:
$$\displaystyle\arg\min_w\|y-Xw\|^2 $$
By finding the optimum of the above equation, I get
$$\displaystyle X^TXw=X^Ty $$
In the case where $...
- 13
1
vote
2
answers
8k
views
Find $\operatorname{Cov}(\hat{\beta}_0, \hat{\beta}_1)$.
Let $Y_1,Y_2,\ldots,Y_n$ and $X_1,X_2,\ldots,X_m$ be random variables with $E(Y_i)=\mu_i$ and $E(X_j)=\xi_j$. Define $$U_1=\sum_{i=1}^n a_i Y_i\quad\text{and}\quad U_2=\sum_{j=1}^m b_j X_j$$ for ...
- 1,727
1
vote
1
answer
479
views
symmetry of Regression line when SDx equals SDy and non intutive results
This question is taken from Freedman
In a certain class, midterm scores average out to $60$ with an SD of $15,$
as do scores on the final. The correlation between midterm scores and
final ...
- 539
0
votes
3
answers
500
views
Derivation of Linear Regression using Normal Equations
I was going through Andrew Ng's course on ML and had a doubt regarding one of the steps while deriving the solution for linear regression using normal equations.
Normal equation: $\theta=(X^TX)^{-1}X^...
- 3
0
votes
0
answers
95
views
Instrumental Variable Regression Questions
We're learning about instrumental variable estimators and I just want to confirm my logic in these answers. Really appreciate the help!
a. True- If this happens and these variables are omitted, it ...
- 161
0
votes
1
answer
2k
views
Convert log model to linear model
I have a log-log model as follows:
ln quality = ln price + predictor_2 + predictor_3
I ran a regression and using the coefficient values obtained, I predicted log quality values and then I plotted ...
- 101
0
votes
1
answer
930
views
Proving $a$ (in $Y = aX + b + e$) satisfies $a = Cov(X, Y )/Var(X)$
In a linear regression model, we postulate that random variables $X$ and $Y$ are related by
$$Y = aX + b + e$$
where a and b are constants (called the regression coefficients) and e (representing ...
- 11
0
votes
1
answer
341
views
ANOVA - Distribution of $\hat{\beta}_1$ still exists although $\beta_1=0$ under $H_0$?
We're doing simple linear regression. Anova decomposition. $$SS=RegSS+RSS$$ or better $$(n-1)s_Y^2=\hat{b}_1^2s_{XX}+(n-2)s^2$$
We know that when the fit is good, then RegSS will be large. Therefore ...
- 5,267
0
votes
1
answer
205
views
Fitting a line to the set of planes
The methods of fitting lines, planes to the set of points are rather popular. But is it possible to do anything similar for the case when the 3D line is fitted to the set of 3D planes?
I.e. there are ...
- 454
0
votes
2
answers
615
views
Matrix Least Squares on the Rows Instead of Columns
I want to solve the equation $AB=C, A\epsilon \mathbb{R}^{3\times 3}, B\epsilon \mathbb{Z}^{3\times 10}, C\epsilon \mathbb{R}^{3\times 10}$ for $A$. The solution should minimize the norm (best is ...
- 99
0
votes
1
answer
59
views
Partial Derivative of $L[\phi]$ with respect to $\phi_0$ and $\phi_1$
Problem 2.1 from Simon Prince's "Understanding Deep Learning".
$L[\phi] = \sum_{i=1}^I (\phi_0 + \phi_1x_i - y_i)^2$
Find $\frac{\partial L}{\partial\phi_0}$ and $\frac{\partial L}{\partial\...
- 41
0
votes
1
answer
3k
views
Linearising a cubic function
I'm not sure if this is the right term, but I want to 'linearise' an equation of the form $y=ax+bx^3$. What I mean is that if I had another function $y=e^x$, then I can plot $\ln(y)$ against $x$ and I ...
- 373
0
votes
1
answer
421
views
Convert Convex $ {L}_{1.5} $ Regression Problem into Semi Definite Program (SDP)
Given the following Linear $ {L}_{1.5} $ Regression Problem:
$$ \arg \min_{x} {\left\| A x - b \right\|}_{1.5} $$
How could one redefine it as Semi Definite Programming Problem?
My (Partial) ...
- 9,057
0
votes
1
answer
649
views
Equivalent formulation of LASSO?
I am currently trying to tell wheter or not those two problems are equivalent :
$$\min_x \|x\|_1 \text { s.t. } \|Ax-y\|^2_2 \le \varepsilon.$$
And
$$\min_x \|Ax-y\|^2_2 \text { s.t. } \|x\|_1\le ...
- 8,284
0
votes
1
answer
70
views
Using linear least-squares to solve this system
I have $\frac{n(n-1)}{2}$ equations of the form:
$$2(x_j-x_i)X+2(y_j-y_i)Y+2(z_j-z_i)Z+2v^2\tau(t_i-t_j)=2v^2(t_i^2-t_j^2)+(x_j^2+y_j^2+z_j^2)-(x_i^2+y_i^2+z_i^2)$$
With $n\geq 4$, and where ...
- 1,381