Questions tagged [linear-regression]
For questions about linear regressions, an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables.
1,322
questions
20
votes
1
answer
36k
views
what is the variance of a constant matrix times a random vector?
$\newcommand{\Var}{\operatorname{Var}}$In this video is claimed that if the equation of errors in OLS is given by:
$$u=y - X\beta$$
Then in the presence of heteroscedasticity the variance of $u$, will ...
- 323
18
votes
3
answers
32k
views
Proof that trace of 'hat' matrix in linear regression is rank of X
I understand that the trace of the projection matrix (also known as the "hat" matrix) X*Inv(X'X)*X' in linear regression is equal to the rank of X. How can we prove that from first principles, i.e. ...
- 283
14
votes
2
answers
10k
views
why is the least square cost function for linear regression convex
I was looking at Andrew Ng's machine learning course and for linear regression he defined a hypothesis function to be $h(x) = \theta_0 + \theta_1x_1 + \dots + \theta_nx_n$, where $x$ is a vector of ...
- 421
12
votes
1
answer
22k
views
Derivative of Mean Squared Error
I'm studying with a book and I'm at the Linear Regression part. The author is showing that we have to calculate the derivative of each part of the equation that leads to the loss.
But he's using the ...
- 263
8
votes
2
answers
17k
views
When can we say that $A^{\mathrm T} B = B^{\mathrm T} A$?
I was looking at the derivation of the normal equation from here.
Now, the author has used the fact that $A^{\mathrm T} B = B^{\mathrm T} A$ to reach the step shown in the below image. Can anyone ...
- 310
7
votes
3
answers
16k
views
Proof of Gauss-Markov theorem
Theorem: Let $Y=X\beta+\varepsilon$ where $$Y\in\mathcal M_{n\times 1}(\mathbb R),$$ $$X\in \mathcal M_{n\times p}(\mathbb R),$$ $$\beta\in\mathcal M_{n\times 1}(\mathbb R ),$$ and $$\varepsilon\in\...
- 1,727
7
votes
1
answer
5k
views
Explain about the Correlation of Error Terms in Linear Regression Models
I would like to ask for the interpretation, both mathematically and intuitively if possible, about the homoscedasticity of the variance of errors in linear regression models.
If there is correlation ...
- 405
7
votes
4
answers
3k
views
Why does $A^TAx = A^Tb$ have infinitely many solution algebraically when $A$ has dependent columns?
This is a problem from least square approximation, where we solve the equation $A^TAx = A^Tb$ when $Ax = b$ is unsolvable.
The case I am dealing with is when A has dependent columns, i.e. A is an m by ...
- 181
7
votes
4
answers
20k
views
Best Fit Line with 3d Points
Okay, I need to develop an alorithm to take a collection of 3d points with x,y,and z components and find a line of best fit. I found a commonly referenced item from Geometric Tools but there doesn't ...
- 203
7
votes
0
answers
534
views
Perturbation theory for least squares for very different A, b
Consider the least squares problem
$f(x;A,b) = \|Ax-b\|_2^2$
and define $x^*$ the minimizer of $f(x;\hat A,\hat b)$, and $\hat x$ the minimizer of $f(x; A_2, b_2)$.
I want to put some bound on $\|...
- 1,846
6
votes
1
answer
840
views
Applying the Normal Equations to solve the Linear Regression Problems.
I am new to machine learning and I am currently studying the gradient descent method and its application for linear regression. An iterative method known as gradient descent is finding the linear ...
- 163
6
votes
1
answer
2k
views
Connection Between Orthogonal Projection onto the Unit Simplex and the Softmax Function
Referring to papers Softmax to Sparsemax and Efficient Projections onto the L1-Ball, what is the relationship between a euclidean projection onto the probability simplex and applying the Softmax ...
- 91
6
votes
1
answer
1k
views
Proving Linear Regression by Using Physical Springs Model
I found a nice proof of the linear regression formulas by using physical springs in Mark Levi's Mathematical Mechanic on page 43.
Linear Regression (The Best Fit) via Springs
Imagine a collection ...
- 2,954
6
votes
1
answer
256
views
Proof of $\frac{1}{n}\mathrm{E} \left[ \| \mathbf{X}\mathbf{\hat{w}} - \mathbf{X}\mathbf{w}^{*} \|^{2}_{2} \right] = \sigma^{2}\frac{d}{n}$
I am trying to find a proof for the MSE of a linear regression:
\begin{gather}
\frac{1}{n}\mathrm{E} \left[ \| \mathbf{X}\mathbf{\hat{w}} - \mathbf{X}\mathbf{w}^{*} \|^{2}_{2} \right] = \sigma^{2}\...
- 73
6
votes
1
answer
3k
views
Bayesian Interpretation for Ridge Regression and the Lasso
I'm learning the book "Introduction to Statistical Learning" and in the Chapter 6 about "Linear Model Selection and Regularization", there is a small part about "Bayesian ...
- 405
6
votes
0
answers
153
views
Prime number intercept
Suppose I arrange my (infinite) list of prime numbers in the following way: \begin{array}{c|c}x_i&2&5&11&17&23&31&\cdots\\\hline y_i&3&7&13&19&29&37&...
- 27.1k
5
votes
3
answers
7k
views
Prove that $R^{2}$ cannot decrease when adding a variable
I know that in general this is true because the smaller model is nested within the larger model, so the larger model must have SSE at least as low as the smaller one, but I'm having a hard time ...
- 533
5
votes
2
answers
1k
views
Regression to the mean - a simple question
In my statistics book there is a following question:
In studies dating back over 100 years, it's well established that regression toward the mean occurs between the heights of fathers and the ...
- 85
5
votes
1
answer
1k
views
Basic exponential regression
Background:
I'm attempting to learn about basic statistics for the infrastructure asset management industry:
Basic math explanation (related to estimating linear regression with no intercept)
...
- 425
5
votes
3
answers
353
views
Why isn't the linear regression coefficient not just the average vector to data points?
I am having trouble intuitively understanding the correctness of the formula to compute the coefficient for the regression line in a linear regression.
I know the formula is
$$\frac{\sum_{i=1}^N (...
- 1,373
5
votes
4
answers
3k
views
Does least squares (approximate solution) minimize the orthogonal distance of $b$ to $Ax$, or does it minimize the error projected along the $b$ axis?
I have always been confused about whether the approximate solution to $Ax=b$ is equivalent to minimizing the average distance of all of the $b$ vectors to $Ax$, or whether it is minimizing the ...
- 1,603
5
votes
2
answers
1k
views
How can I get the gradient of the normal equation for weighted linear regression?
The normal equation for weighted linear regression looks like this:
$$J(\theta) = (X\theta - y)^TW(X\theta - y),$$ where $X\in\Re^{m\times n}$, $\theta\in\Re^{n\times n}$, $y\in\Re^{m\times 1}$, $W\...
- 209
5
votes
1
answer
4k
views
Variance of Beta in the Normal Linear Regression Model
Let $Y_1, Y_2, \ldots, Y_n$ represent response variables and let $x_1, x_2,\ldots, x_n$ be the associated explanatory variables.
In the normal linear regression model, it's assumed that:
$$Y_i \sim ...
- 635
5
votes
1
answer
2k
views
Least Absolute Deviation (LAD) Line Fitting / Regression
I want to implement robust line fitting over a set of $n$ points $(x_i,y_i)$ by means of the Least Absolute Deviation method, which minimizes the sum
$$\sum_{i=1}^n |y_i-a-bx_i|.$$
As described for ...
user65203
5
votes
1
answer
5k
views
Does Least Squares Regression Minimize the RMSE?
In the application of least-squares regression to data fitting, the quantity of minimization is the sum of squares (sum of squared errors, to be specific). I believe this fitting also minimizes the ...
- 127
5
votes
2
answers
846
views
Prove uniqueness of solutions of different OLS matrix cases
Let $D = \{(x_1, y_2), (x_2, y_2), \ldots , (x_n, y_n)\}$ where $x_i \in \mathbb{R}^d$ and $y_i \in \mathbb{R}$. One may use linear regression to predict $y$ as $w^Tx$ for some parameter vector $w \in ...
- 105
5
votes
1
answer
154
views
standard error does not change with standardization in OLS
I want to show that in a simple linear regression ($y = X\beta + \epsilon$), when the mean is subtracted from any feature (i.e. some column of the matrix $X$), the t-statistics of all the non-...
- 1,870
5
votes
0
answers
217
views
Dimension free gram matrix inner product
Let $\{x_i\}_{i = 1}^n$ be $n$ vectors of $d$ dimesnions.
We stack each $x_i$ as a row vector to form a matrix $X$ of dimension $\mathbb{R}^{n\times d}$
Let $\{y_i\}_{i = 1}^n$ be scalars (say all are ...
- 487
5
votes
0
answers
95
views
Convergence speed of discrete approximation
Here I asked the question about approximating the function $g(x) := \mathbb{E}(f(x,Y))$, where $x \in R$ and $Y$ is a random variable. If you follow the link you will see that $g(x)$ can be ...
- 319
4
votes
1
answer
2k
views
Why is the sum of elements in each row of $X(X^T X)^{-1} X^T$ in OLS equal to $1$?
I have noticed that the sum of elements in each row (and each column, since the matrix is symmetric) of $X(X^T X)^{-1}X^T$, where $X$ is the information matrix in the OLS regression, is equal to 1. Is ...
- 583
4
votes
4
answers
1k
views
Does $(X'X)^{-1}$ always exist?
I'm studing Machine Learning theory and I have a questions about Normal Equation. Normal Equation is:
$\Theta = (X'X)^{-1}X'Y\tag 1$
I now that ( in some cases) we can use this other equation:
$\Theta ...
4
votes
2
answers
777
views
Linear Least Squares with Monotonicity Constraint
I'm interested in the multidimensional linear least squares problem: $$\min_{x}||Ax-b||^2$$
subject to a monotonicity constraint for $x$, meaning that the elements of $x$ are monotonically increasing: ...
- 51
4
votes
1
answer
2k
views
Variance Estimate in linear regression
In a linear regression, $y=X\beta+\epsilon$, where $\epsilon\sim N(0, \sigma^2)$, $X\sim R^{N \times (p+1)}$. Assume the observations $y_i$ are uncorrelated and have constant variance $\sigma^2$, and ...
- 51
4
votes
1
answer
48
views
Why does $2$ appear in $95\%$ confidence intervals?
I was told that, in linear regression, the $95\%$ CI for $\beta$ is $$\beta \in \left(\hat\beta-2 \sigma,\hat\beta+2\sigma\right), \text{ where } \sigma = \text{standard error}(\hat\beta).$$
My ...
4
votes
1
answer
622
views
(Linear Regression) Proving that Linear Regression is linear invariant
I want to ask how to show that Linear Regression is linear invariant? The problem is specified in the following picture:
Here is the "solution" for the problem. But I really get confused by its ...
4
votes
2
answers
3k
views
Proving Convergence of Least Squares Regression with i.i.d. Gaussian Noise
I have a basic question that I can't seem to find an answer for -- perhaps I'm not wording it correctly. Suppose that we have an $n$-by-$d$ matrix, $X$ that represents input features, and we have a $n$...
- 189
4
votes
1
answer
6k
views
If $Y=X\beta+\epsilon$, prove that the least square estimator $\hat\beta$ is independent of $Y-X\hat{\beta}$
Let $Y=X\beta+\epsilon$, where $Y$ is an $n$ by $1$ vector, $X$ is an $n$ by $p$ matrix with full rank and $\epsilon$ is an $n$ by 1 vector of random errors independently and normally distribution ...
- 1,306
4
votes
2
answers
86
views
Line of best fit for $\{(n,n+\sin n) : n \in \mathbb{Z}\}$
It seems intuitive that the line of best fit for $\{(n,n+\sin n) : n\in \mathbb{Z}\}$ should be $y=x$.
More concretely, it seems like a reasonable conjecture would be:
If $y = m_k x + b_k$ is the ...
- 4,333
4
votes
2
answers
1k
views
Proof of Batch Gradient Descent's cost function gradient vector
In the book Hands-On Machine Learning with Scikit-Learn & TensorFlow, the author only showed the formula for the Batch Gradient Descent method, such as:
$ \dfrac{\partial}{\partial \theta_{j}} ...
4
votes
2
answers
4k
views
Confusion in Relationship between regression line slope and covariance
In simple linear regression model between RVs $(X,Y)$, the slope $\hat\beta_1$ is given as
$$
\hat\beta_1 = \dfrac{\sum_i^N(x-\overline{x})(y - \overline{y})}{\sum_i^N(x - \overline{x})^2} \tag{1}
$$
...
4
votes
2
answers
8k
views
In linear regression, why is the hat matrix idempotent, symmetric, and p.s.d.?
In linear regression,
$$y = X \beta + \epsilon$$
where $y$ is a $n \times 1$ vector of observations for the response variable,
$X = (x_{1}^{T}, ..., x_{n}^{T}), x_{i} \in \mathbb{R}^p. i =1,...,n$...
- 1,235
4
votes
2
answers
490
views
How to fit an ODE to data?
Consider the following ODE
$$
y'(t)=\alpha x(t)-\beta y(t)
$$
and the following datasets
$$
X=\{(t_0,x_0),...,(t_n,x_n)\}\\
Y=\{(t_0,y_0),...,(t_n,y_n)\}
$$
How can I find $\alpha$ and $\beta$ that ...
- 3,465
4
votes
1
answer
1k
views
Basic application of category theory to data science
Linear regression is the algorithm that, given a set of vectors ${\bf x}_i \in \mathbb{R}^p$, and a set of targets $y_i \in \mathbb{R}$, returns a vector ${\bf w} \in \mathbb{R}^p$ minimizing the ...
- 648
4
votes
1
answer
272
views
Does quadratic risk of MLE for multivariate linear regression go to zero with more and more data?
For the simple multivariate linear regression with Gaussian noise: $\mathbf{Y} = \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\epsilon}$, where
$\mathbf{Y} \in \mathbb{R}^n$: the vector of dependent ...
- 1,527
4
votes
1
answer
379
views
Linear Regression quadratic terms
I have a hard time understanding the term 'linear regression'. For what I know, linear means polynomial of degree 1. But then, I found that in one of my lectures, the lecturers are saying that this ...
- 158
4
votes
1
answer
130
views
Sqrt LASSO vs LASSO
In the paper Square Root Lasso: Pivotal Recovery of Sparse Signals via Conic Programming they talk about Sqrt-LASSO which is simply just trying to minimize $\|Ax-b\|_2 + \lambda\|x\|_1$ rather than ...
- 63
4
votes
1
answer
224
views
How to check the consistency of OLS estimator in macroeconomic models
Problem:
We have a model $$C_t = a + b Y_t + e_t$$ and $$ Y_t = C_t + I_t$$ It's known that $Cov(I, e)$ is zero.
A student estimates the following model: $$C_t = a + b Y_t + e_t$$
Are the estimators $\...
- 422
4
votes
2
answers
554
views
Correlation coefficient and regression line : Geometric intuition
correlation coefficient
$$r = \frac{1}{n}\sum_{i=1}^n\frac{(x_i-\bar x)(y_i-\bar y)}{\sigma_x\cdot\sigma_y}$$
may be thought of as cosine of angle between two $n$-dimensional vectors
$$ (x_1- \bar ...
- 539
4
votes
1
answer
250
views
Why is $E[(Y-f(X))^2] = E[(Y-f(X))^T(Y-f(X))]$ in derivation of the regression function f?
I am working through Chapter 2 of The Elements of Statistical Learning (Hastie, Tibshirani, Friedman). I have problems following the authors as they analytically derive the regression coefficients $\...
- 147
4
votes
2
answers
199
views
intuition behind having a unique regression line
I understand this mathematically. we have function of 2 variables represents the sum of square errors. We have to find the $a$ and $b$ that minimize the function. there is only one minimum point.
But ...
- 1,573