# Questions tagged [linear-regression]

For questions about linear regressions, an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables.

1,322 questions
Filter by
Sorted by
Tagged with
36k views

### what is the variance of a constant matrix times a random vector?

$\newcommand{\Var}{\operatorname{Var}}$In this video is claimed that if the equation of errors in OLS is given by: $$u=y - X\beta$$ Then in the presence of heteroscedasticity the variance of $u$, will ...
• 323
32k views

### Proof that trace of 'hat' matrix in linear regression is rank of X

I understand that the trace of the projection matrix (also known as the "hat" matrix) X*Inv(X'X)*X' in linear regression is equal to the rank of X. How can we prove that from first principles, i.e. ...
• 283
10k views

### why is the least square cost function for linear regression convex

I was looking at Andrew Ng's machine learning course and for linear regression he defined a hypothesis function to be $h(x) = \theta_0 + \theta_1x_1 + \dots + \theta_nx_n$, where $x$ is a vector of ...
• 421
22k views

### Derivative of Mean Squared Error

I'm studying with a book and I'm at the Linear Regression part. The author is showing that we have to calculate the derivative of each part of the equation that leads to the loss. But he's using the ...
• 263
17k views

### When can we say that $A^{\mathrm T} B = B^{\mathrm T} A$?

I was looking at the derivation of the normal equation from here. Now, the author has used the fact that $A^{\mathrm T} B = B^{\mathrm T} A$ to reach the step shown in the below image. Can anyone ...
• 310
16k views

• 1,373
3k views

### Does least squares (approximate solution) minimize the orthogonal distance of $b$ to $Ax$, or does it minimize the error projected along the $b$ axis?

I have always been confused about whether the approximate solution to $Ax=b$ is equivalent to minimizing the average distance of all of the $b$ vectors to $Ax$, or whether it is minimizing the ...
• 1,603
1k views

The normal equation for weighted linear regression looks like this: $$J(\theta) = (X\theta - y)^TW(X\theta - y),$$ where $X\in\Re^{m\times n}$, $\theta\in\Re^{n\times n}$, $y\in\Re^{m\times 1}$, $W\... • 209 5 votes 1 answer 4k views ### Variance of Beta in the Normal Linear Regression Model Let$Y_1, Y_2, \ldots, Y_n$represent response variables and let$x_1, x_2,\ldots, x_n$be the associated explanatory variables. In the normal linear regression model, it's assumed that: $$Y_i \sim ... • 635 5 votes 1 answer 2k views ### Least Absolute Deviation (LAD) Line Fitting / Regression I want to implement robust line fitting over a set of n points (x_i,y_i) by means of the Least Absolute Deviation method, which minimizes the sum$$\sum_{i=1}^n |y_i-a-bx_i|.$$As described for ... 5 votes 1 answer 5k views ### Does Least Squares Regression Minimize the RMSE? In the application of least-squares regression to data fitting, the quantity of minimization is the sum of squares (sum of squared errors, to be specific). I believe this fitting also minimizes the ... • 127 5 votes 2 answers 846 views ### Prove uniqueness of solutions of different OLS matrix cases Let D = \{(x_1, y_2), (x_2, y_2), \ldots , (x_n, y_n)\} where x_i \in \mathbb{R}^d and y_i \in \mathbb{R}. One may use linear regression to predict y as w^Tx for some parameter vector w \in ... • 105 5 votes 1 answer 154 views ### standard error does not change with standardization in OLS I want to show that in a simple linear regression (y = X\beta + \epsilon), when the mean is subtracted from any feature (i.e. some column of the matrix X), the t-statistics of all the non-... • 1,870 5 votes 0 answers 217 views ### Dimension free gram matrix inner product Let \{x_i\}_{i = 1}^n be n vectors of d dimesnions. We stack each x_i as a row vector to form a matrix X of dimension \mathbb{R}^{n\times d} Let \{y_i\}_{i = 1}^n be scalars (say all are ... • 487 5 votes 0 answers 95 views ### Convergence speed of discrete approximation Here I asked the question about approximating the function g(x) := \mathbb{E}(f(x,Y)), where x \in R and Y is a random variable. If you follow the link you will see that g(x) can be ... 4 votes 1 answer 2k views ### Why is the sum of elements in each row of X(X^T X)^{-1} X^T in OLS equal to 1? I have noticed that the sum of elements in each row (and each column, since the matrix is symmetric) of X(X^T X)^{-1}X^T, where X is the information matrix in the OLS regression, is equal to 1. Is ... • 583 4 votes 4 answers 1k views ### Does (X'X)^{-1} always exist? I'm studing Machine Learning theory and I have a questions about Normal Equation. Normal Equation is: \Theta = (X'X)^{-1}X'Y\tag 1 I now that ( in some cases) we can use this other equation: \Theta ... 4 votes 2 answers 777 views ### Linear Least Squares with Monotonicity Constraint I'm interested in the multidimensional linear least squares problem:$$\min_{x}||Ax-b||^2$$subject to a monotonicity constraint for x, meaning that the elements of x are monotonically increasing: ... • 51 4 votes 1 answer 2k views ### Variance Estimate in linear regression In a linear regression, y=X\beta+\epsilon, where \epsilon\sim N(0, \sigma^2), X\sim R^{N \times (p+1)}. Assume the observations y_i are uncorrelated and have constant variance \sigma^2, and ... 4 votes 1 answer 48 views ### Why does 2 appear in 95\% confidence intervals? I was told that, in linear regression, the 95\% CI for \beta is$$\beta \in \left(\hat\beta-2 \sigma,\hat\beta+2\sigma\right), \text{ where } \sigma = \text{standard error}(\hat\beta).$$My ... 4 votes 1 answer 622 views ### (Linear Regression) Proving that Linear Regression is linear invariant I want to ask how to show that Linear Regression is linear invariant? The problem is specified in the following picture: Here is the "solution" for the problem. But I really get confused by its ... 4 votes 2 answers 3k views ### Proving Convergence of Least Squares Regression with i.i.d. Gaussian Noise I have a basic question that I can't seem to find an answer for -- perhaps I'm not wording it correctly. Suppose that we have an n-by-d matrix, X that represents input features, and we have a n... 4 votes 1 answer 6k views ### If Y=X\beta+\epsilon, prove that the least square estimator \hat\beta is independent of Y-X\hat{\beta} Let Y=X\beta+\epsilon, where Y is an n by 1 vector, X is an n by p matrix with full rank and \epsilon is an n by 1 vector of random errors independently and normally distribution ... • 1,306 4 votes 2 answers 86 views ### Line of best fit for \{(n,n+\sin n) : n \in \mathbb{Z}\} It seems intuitive that the line of best fit for \{(n,n+\sin n) : n\in \mathbb{Z}\} should be y=x. More concretely, it seems like a reasonable conjecture would be: If y = m_k x + b_k is the ... • 4,333 4 votes 2 answers 1k views ### Proof of Batch Gradient Descent's cost function gradient vector In the book Hands-On Machine Learning with Scikit-Learn & TensorFlow, the author only showed the formula for the Batch Gradient Descent method, such as: \dfrac{\partial}{\partial \theta_{j}} ... 4 votes 2 answers 4k views ### Confusion in Relationship between regression line slope and covariance In simple linear regression model between RVs (X,Y), the slope \hat\beta_1 is given as$$ \hat\beta_1 = \dfrac{\sum_i^N(x-\overline{x})(y - \overline{y})}{\sum_i^N(x - \overline{x})^2} \tag{1} $$... 4 votes 2 answers 8k views ### In linear regression, why is the hat matrix idempotent, symmetric, and p.s.d.? In linear regression,$$y = X \beta + \epsilon$$where y is a n \times 1 vector of observations for the response variable, X = (x_{1}^{T}, ..., x_{n}^{T}), x_{i} \in \mathbb{R}^p. i =1,...,n... • 1,235 4 votes 2 answers 490 views ### How to fit an ODE to data? Consider the following ODE$$ y'(t)=\alpha x(t)-\beta y(t) $$and the following datasets$$ X=\{(t_0,x_0),...,(t_n,x_n)\}\\ Y=\{(t_0,y_0),...,(t_n,y_n)\} $$How can I find \alpha and \beta that ... • 3,465 4 votes 1 answer 1k views ### Basic application of category theory to data science Linear regression is the algorithm that, given a set of vectors {\bf x}_i \in \mathbb{R}^p, and a set of targets y_i \in \mathbb{R}, returns a vector {\bf w} \in \mathbb{R}^p minimizing the ... • 648 4 votes 1 answer 272 views ### Does quadratic risk of MLE for multivariate linear regression go to zero with more and more data? For the simple multivariate linear regression with Gaussian noise: \mathbf{Y} = \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\epsilon}, where \mathbf{Y} \in \mathbb{R}^n: the vector of dependent ... • 1,527 4 votes 1 answer 379 views ### Linear Regression quadratic terms I have a hard time understanding the term 'linear regression'. For what I know, linear means polynomial of degree 1. But then, I found that in one of my lectures, the lecturers are saying that this ... • 158 4 votes 1 answer 130 views ### Sqrt LASSO vs LASSO In the paper Square Root Lasso: Pivotal Recovery of Sparse Signals via Conic Programming they talk about Sqrt-LASSO which is simply just trying to minimize \|Ax-b\|_2 + \lambda\|x\|_1 rather than ... 4 votes 1 answer 224 views ### How to check the consistency of OLS estimator in macroeconomic models Problem: We have a model$$C_t = a + b Y_t + e_t$$and$$ Y_t = C_t + I_t$$It's known that Cov(I, e) is zero. A student estimates the following model:$$C_t = a + b Y_t + e_t$$Are the estimators \... • 422 4 votes 2 answers 554 views ### Correlation coefficient and regression line : Geometric intuition correlation coefficient$$r = \frac{1}{n}\sum_{i=1}^n\frac{(x_i-\bar x)(y_i-\bar y)}{\sigma_x\cdot\sigma_y}$$may be thought of as cosine of angle between two n-dimensional vectors$$ (x_1- \bar ... • 539 4 votes 1 answer 250 views ### Why is$E[(Y-f(X))^2] = E[(Y-f(X))^T(Y-f(X))]$in derivation of the regression function f? I am working through Chapter 2 of The Elements of Statistical Learning (Hastie, Tibshirani, Friedman). I have problems following the authors as they analytically derive the regression coefficients$\...
• 147
I understand this mathematically. we have function of 2 variables represents the sum of square errors. We have to find the $a$ and $b$ that minimize the function. there is only one minimum point. But ...