Questions tagged [linear-regression]
For questions about linear regressions, an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables.
865
questions
0
votes
0answers
6 views
Conditional Covariance is zero in classical linear regression model
I just read that in the classical linear regression model (Y=Xβ+ε) the Cov(β ̂,ε ̂│X)=0.
How can we derive this fact?
What is clear is that if X and Y are independent, then Cov(X,Y)=0.
Also, for any ...
1
vote
0answers
20 views
How to write residual in terms of error in simple linear regression
It's given that $y_{i}=\beta_{0}+\beta_{1} x_{i}+\epsilon_{i}, i=1,2, \ldots, n, E\left(\epsilon_{i} \mid X\right)=0, \operatorname{var}\left(\epsilon_{i} \mid X\right)=\sigma^{2},$ for all $i$
Can I ...
0
votes
0answers
17 views
One sided test of coefficients in Simple linear regression
Let $Y_{i} = \beta_{0} + \beta_{1}X_{i} + \epsilon_{i}$ be a simple linear regression model with independent errors and iid normal distribution.
I have done two sided t-test for coefficients by test ...
0
votes
0answers
10 views
Gauss-Markov Theorem proof
Is it possible to prove this part of the Gauss-Markov Theorem:
w'β ̂ is BLUE (best linear unbiased estimator) for w'β, where β ̂ is the OLS estimate of β, and w is a nonzero vector.
I know how to ...
0
votes
0answers
25 views
How to proof a test statistics is pivotal
Consider we have a classic normal linear regression model
$$y_t = X_t\beta + u_t, u \sim NID(0,\sigma^2) $$
Where we define n observations, $\beta$ is a k-vector and 1 x k vector of regressors $X_t$ ...
1
vote
0answers
13 views
Linear regression feature is combination of other features, but has OPPOSITE correlation?
We're examining "salesperson cost per sale," where cost is just what the agent is paid in compensation. In other words, our target variable is the ratio of (salesperson pay)/(count of sales)....
0
votes
0answers
15 views
Regression based on function recurrence
There is a hypothetical machine which takes an integer $x$ and returns an integer $y$ such that $y=F(x)+\varepsilon$ where $\varepsilon$ is an integer.
It is known that the function is of the form
$F(\...
0
votes
0answers
8 views
What is the benefit of lambda over quadratic lambda in linear regression regularization
For linear regression, I got this regularized least mean errors
\begin{align}
\mathcal{L}_{A, \Lambda}(\theta)_1 & = ||y - X\theta||^2_{A} + ||\theta||^2_{\Lambda} \\
\mathcal{L}_{A, \Lambda}(\...
0
votes
0answers
5 views
Is it possible for out-of-sample residuals to be much smaller than in-sample ones?
When doing linear regression the classic trap is to report residuals on the training data rather than testing data.
However I seem to be consistently getting significantly lower errors on my test data ...
0
votes
0answers
24 views
Ways to describe matrix whose elements are linear combinations of elements of a vector?
The motivation for this question comes from trying to solve the following system:
$$B = AX: \left[X\right]_{ij}=\sum_{l=1}^k \alpha_{k}^{(ij)}c_k$$
Where $A,B \in \mathbb{R}^{N\times p}, \hspace{0.1in}...
0
votes
0answers
6 views
What is random design in local constant estimation?
Suppose we have paired observations $\left(x_{i}, Y_{i}\right)$ and assume the nonparametric model
$$
Y_{i}=f\left(x_{i}\right)+\epsilon_{i}, \quad i=1, \ldots, n
$$
where $Y_{i}$ 's are responses, ...
0
votes
0answers
6 views
What is a relationship between standard deviation of a dependent variable and standard deviation of residuals?
In a book, ThinkStats, by Allen Downey, it uses an example of a simple linear regression model to predict babies'' birth weights using mothers' ages. Then, it mentions that standard deviation of the ...
0
votes
0answers
12 views
Is the formula for standard error for the slope of a linear regression with intercept the same as without?
If we are given sets $X$ and $Y$. The standart error formula for $\alpha$ coefficient of the regrssion $\hat{y} = \alpha x + \beta$ is $$ \frac{\sum{(y_i -\hat{y})^2}/(n-2)}{\sqrt{\sum(x_i-\bar{x})^2}}...
0
votes
0answers
18 views
Does the equality coefficients of linear regression of X onto Y and Y onto X imply coincidence of the lines?
Let's assume that we consider the model without an intercept $\hat{y_i} = x_i\hat{\beta}$. So, the formula for $\hat{\beta}$ is $\frac{ \sum\limits_{i=1}^n x_i y_i}{\sum\limits_{j=1}^n x_j^2}$. I ...
0
votes
0answers
8 views
Solving Matrix Equation in Residual Sum of Squares when Inputs not Linearly Independent
I'm reading Elements of Statistical Learning. On page 46, it talks about the residual sum of squares.
We want to minimize the residual sum of squares.
Let $X$ be a $N \times p+1$ matrix. Let $\beta$ ...
0
votes
1answer
22 views
Are $\beta_0$ and $\beta_1$ unbiased estimators of $\hat\beta_0$ and $\hat\beta_1$?
When we are discussing simple linear regression with:
$$Y_i = \beta_0 + X_i\beta_1 +u_i$$
$\hat\beta_0$ and $\hat\beta_1$ are estimates of this model using OLS.
With a simple proof we get $E(\hat\...
3
votes
1answer
29 views
Does quadratic risk of MLE for multivariate linear regression go to zero with more and more data?
For the simple multivariate linear regression with Gaussian noise: $\mathbf{Y} = \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\epsilon}$, where
$\mathbf{Y} \in \mathbb{R}^n$: the vector of dependent ...
0
votes
0answers
15 views
Most Important Variables Linear Regression
When determining the most important variables in linear regression, I understand that first we need to decrease multicollinearity, if present, by using some sort of variable reduction method such as ...
0
votes
0answers
21 views
Confidence Interval of linear regression model - Standard Deviation vs Standard Error
For a (large) experimental dataset I am assuming a quadratic model and perform a regression analysis. Then I'd like to have some information about the accuracy of my model parameters, so I'm ...
0
votes
0answers
8 views
Derivation of multiple linear regression model, the 'matrix' problem.
I'm studying multiple linear regression.
There're so many posts about the derivation of formula. But I can't find the one fully explaining how to deal with the matrix. I'm not good at linear algebra ...
2
votes
2answers
99 views
Regression linear model without error term
I have this linear model from a regression:
$Y_i$ = $\beta_1X_{i1}$ + ... + $\beta_mX_{im}$ + $\epsilon_i$
The matrix representation is:
$Y$ = $X\beta$ + $\epsilon$
In a lot of places like Wikipedia ...
1
vote
0answers
71 views
Covariance between estimated y and observed y(in simple linear regression)
I wonder what is $Cov(y,\hat{y})$ the Covariance between estimated $\hat{y}$ and observed value $y$, in the situation of simple linear regression.
Some Youtube tutor said it's 'O', but i can't agree ...
0
votes
0answers
11 views
How to decode/understand the math behind ACF and PACF?
For the past month I have been trying to understand the math behind the autocorrelation function and partial autocorrelation function for time-series project I have been working on. However, I am only ...
1
vote
0answers
48 views
Continuity between Polynomial Fits
I'm currently working on a polynomial regression project and I've hit a wall. I'm trying to fit time-series data where I wish to "scroll" a window across the data and fit a polynomial to the ...
0
votes
0answers
37 views
Gradient descent's cost function: Mean Squared Error vs. Sum of Squared Errors
In many introductory Machine Learning textbooks or online resources, the cost function to be optimized with gradient descent to find a linear regression model is the Mean Squared Error (MSE), defined ...
0
votes
1answer
42 views
Distribution of adjusted R squared under null
Assume the standard linear regression model
\begin{equation}
Y = \beta_0 + \beta_1 X_1 + \cdots + \beta_k X_k + \epsilon
\end{equation}
Under $H_0: \beta_1 = \beta_2 = \cdots = \beta_k = 0$, we know $...
0
votes
0answers
23 views
SVD of a matrix storing gradients of a least squares regression problem
Suppose we have $n$ linear and noisy measurements of the form $y_i = x_i^\top\beta^\ast+e_i$ from an unknown parameter $\beta^\ast\in \mathbb{R}^d$. For each measurement consider the $\ell_2$ loss $...
0
votes
1answer
19 views
Conditions for $\beta_0=\bar{y}$ in simple linear regression using least squares
If we consider a linear model
$\mathbf{y}=\beta_0\mathbf{1}+\beta_1\mathbf{a_1}+\beta_0\mathbf{a_2}+\varepsilon$
with $a_1,a_2$ column vectors with $n$ entries and
$\beta_1\neq\beta_2\neq 0$.
Then, ...
0
votes
0answers
15 views
What are diagonal weighted matrices in Total Least-Square Approach
I am going through Total least square and came across two diagonal matrices $D$ and $T$
$$ D= \operatorname{diag}(d_1, ...,d_m)$$
$$ T = \operatorname{diag}(t_1,...,t_{n+k})$$
Total Least Square
$$(A+ ...
0
votes
1answer
23 views
Why does $\frac{\delta}{\delta\beta}y^TX\beta=\frac{\delta}{\delta\beta}B^TX^Ty?$
Why does $\frac{\delta}{\delta\beta}y^TX\beta=\frac{\delta}{\delta\beta}B^TX^Ty?$
In linear regression the parameters to the function $y=X\beta + \epsilon$ can be found by calculating the derivative ...
0
votes
1answer
29 views
Proof that expected value of estimators for Ordinary Least Squares estimators equals the optimal solution?
I'm doing a linear regression master's course right now and the prof wrote "Properties of OLS estimators is that they are unbiased estimators:
$$\mathbb{E}(\hat{B})=\mathbb{E}[(X^TX)^{-1}X^Ty]$$
$...
0
votes
0answers
17 views
Show that the model with highest $AIC$ is the model with the lowest Mallows $C_p$ statistic.
$AIC = l_S - |S|, C_p = \hat{R_{tr}(S)} + 2|S|\hat{\sigma}^2$, $|S|$-is the number of the columns in design matrix, $\hat{R_{tr}(S)} = \sum_{i=1}^{n}(y-\hat{y}(S))^2$.
Assume a linear regression model ...
1
vote
1answer
33 views
Transformation of function confusion
I am a bit confused about a concept that was recently covered in my stats class. My professor said that there's never any real need to do nonlinear regression, because a function can always be ...
1
vote
2answers
34 views
Ridge regression estimator in high-dimensions: is $(X^TX + \epsilon I_p)^{-1}X^Ty$ finite as $\epsilon \rightarrow 0$?
Consider the ridge regression estimator
$$\hat{\beta}_{\epsilon} := (X^TX + \epsilon I_p)^{-1}X^Ty$$
where $X$ is an $n$ by $p$ matrix with $n < p$. Let $\| \hat{\beta}_{\epsilon} \|_{1} := \sum_{j=...
0
votes
1answer
21 views
Can we estimate $ y = \beta_1\exp(x) + \beta_2\exp(-x) + \beta_3$ using Linear Regression
Can we estimate the following relationship using linear regression. Here, $\beta_1, \beta_2 $ and $\beta_3$ are parameters.
$$ y = \beta_1\exp(x) + \beta_2\exp(-x) + \beta_3$$
0
votes
0answers
29 views
Intuition about a matrix multiplication equality
I encountered the following formula while studying the analytical solution of the Linear regression problem in Machine learning context (Optimizing the weight w.r.t squared error)
$w^{T}X^{T}y=y^{T}Xw$...
-2
votes
1answer
86 views
Coefficient of Determination and Standard Error of the Model
Background explaining standard concepts and standard terminology used in linear regression and analysis of variance:
It will be supposed that one has data points $(X_i,Y_i),\, i = 1,\ldots,n.$
The ...
0
votes
0answers
14 views
Closed form solution for Restricted Weighted Least Squares
From Greene, we know that the closed-form solution of a restricted least squares is:
$\beta_{Constrained} = \beta_{Uncon} - (X'X)^{-1}R'[R(X'X)^{-1}R']^{-1}(R\beta_{Uncon}-r)$.
Is there any similar ...
0
votes
0answers
22 views
Is there an analytical solution to MLE of linear regression with non-normal(exponential) error term?
I know that MLE of normal linear regression $y = k_1x_1+k_2x_2 + \epsilon, \epsilon\sim N(0,1)$ has a nice analytical solution. But what if
the error term is exponential distribution?
the error term ...
5
votes
5answers
180 views
Does $(X'X)^{-1}$ always exist?
I'm studing Machine Learning theory and I have a questions about Normal Equation. Normal Equation is:
$\Theta = (X'X)^{-1}X'Y\tag 1$
I now that ( in some cases) we can use this other equation:
$\Theta ...
0
votes
1answer
12 views
Matrix algebra in least squares regression
Consider this formula for least-squares regression:
\begin{aligned}L(D,{\vec {\beta }})&=||X{\vec {\beta }}-Y||^{2}\\&=(X{\vec {\beta }}-Y)^{T}(X{\vec {\beta }}-Y)\\&=\color{red}{Y^{T}Y-Y^{...
-2
votes
1answer
63 views
Simple Linear Regression Machine Learning Course [closed]
Your friend in the U.S. gives you a simple regression fit for predicting house prices from square feet. The estimated intercept is -44850 and the estimated slope is 280.76. You believe that your ...
0
votes
1answer
44 views
Find rank of matrix [closed]
If $X$ is a $N \times D$ matrix with $(D\gg N)$ with $\operatorname{rank}(X) = N$, what is $\operatorname{rank}(X^T \cdot X)$ where $X^T$ is the transpose matrix of $X$?
I am little new to linear ...
1
vote
1answer
33 views
Soft-EM: E-step for fitting mixed linear regression model
I want to derive the formulas for the soft EM algorithm for the following model $P[y_i | x_i, \pi_{1,\dots,m}, a_{1,\dots,m}] = \sum_{j=1}^m \pi_j \frac{1}{\sqrt{2\pi}\sigma} exp(-\frac{(a_j^T x_i - ...
2
votes
0answers
31 views
Linear regression with non-fixed regressors and some properties
I was talking to my teacher the other day about the OLS and linear regression model $Y = \beta X + \varepsilon$. If the regressors X are fixed numbers and I don't have the normal condition on the ...
1
vote
0answers
26 views
Consistency of least squares for linear regression
Consider the generative model for linear regression w.r.t. the true parameter $w^* \in S^{d-1}$ $$y=Xw^*+e$$ with i.i.d. Gaussian error $e \sim N(0, \sigma^2I_n)$. Let $X \in \mathbb{R}^{n\times d}$ ...
0
votes
1answer
20 views
Coefficient Estimators of $\frac{1}{x^{2}}$ Weighted Least Squares Linear Regression
I have a feeling there should be a mathematical formular for determining the estimators of the coefficients of a $\frac{1}{x^{2}}$ Weighted Linear Regression.
I was able to derive the estimators ($a$ ...
1
vote
1answer
20 views
How can we interpret residual plot in case we have many variables?
In Residual plots, we try to visualize & interpret whether linearity is valid or not in the linear regression model. One way to do this is to plot error term wrt to the independent variable(say x)....
0
votes
0answers
30 views
What do the Eigenvalues and Eigenvectors Of A Coefficient Matrix Tell You?
I see eigenvalues and eigenvectors talked about a lot in relation to a coefficient matrix (in regression and time series models). What is the significance of eigenvalues and eigenvectors in this ...
1
vote
0answers
34 views
Confidence Interval for E(Y0) - Linear Regression
I'm solving exercises as preparation for a statistics exam and I'm scratching my head on the following exercise. I'll start by giving the data I have to work with:
n = 10;
x (average) = 14;
y (average)...
