Questions tagged [linear-regression]

For questions about linear regressions, an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables.

Filter by
Sorted by
Tagged with
0
votes
0answers
6 views

Conditional Covariance is zero in classical linear regression model

I just read that in the classical linear regression model (Y=Xβ+ε) the Cov(β ̂,ε ̂│X)=0. How can we derive this fact? What is clear is that if X and Y are independent, then Cov(X,Y)=0. Also, for any ...
1
vote
0answers
20 views

How to write residual in terms of error in simple linear regression

It's given that $y_{i}=\beta_{0}+\beta_{1} x_{i}+\epsilon_{i}, i=1,2, \ldots, n, E\left(\epsilon_{i} \mid X\right)=0, \operatorname{var}\left(\epsilon_{i} \mid X\right)=\sigma^{2},$ for all $i$ Can I ...
0
votes
0answers
17 views

One sided test of coefficients in Simple linear regression

Let $Y_{i} = \beta_{0} + \beta_{1}X_{i} + \epsilon_{i}$ be a simple linear regression model with independent errors and iid normal distribution. I have done two sided t-test for coefficients by test ...
0
votes
0answers
10 views

Gauss-Markov Theorem proof

Is it possible to prove this part of the Gauss-Markov Theorem: w'β ̂ is BLUE (best linear unbiased estimator) for w'β, where β ̂ is the OLS estimate of β, and w is a nonzero vector. I know how to ...
0
votes
0answers
25 views

How to proof a test statistics is pivotal

Consider we have a classic normal linear regression model $$y_t = X_t\beta + u_t, u \sim NID(0,\sigma^2) $$ Where we define n observations, $\beta$ is a k-vector and 1 x k vector of regressors $X_t$ ...
1
vote
0answers
13 views

Linear regression feature is combination of other features, but has OPPOSITE correlation?

We're examining "salesperson cost per sale," where cost is just what the agent is paid in compensation. In other words, our target variable is the ratio of (salesperson pay)/(count of sales)....
0
votes
0answers
15 views

Regression based on function recurrence

There is a hypothetical machine which takes an integer $x$ and returns an integer $y$ such that $y=F(x)+\varepsilon$ where $\varepsilon$ is an integer. It is known that the function is of the form $F(\...
0
votes
0answers
8 views

What is the benefit of lambda over quadratic lambda in linear regression regularization

For linear regression, I got this regularized least mean errors \begin{align} \mathcal{L}_{A, \Lambda}(\theta)_1 & = ||y - X\theta||^2_{A} + ||\theta||^2_{\Lambda} \\ \mathcal{L}_{A, \Lambda}(\...
0
votes
0answers
5 views

Is it possible for out-of-sample residuals to be much smaller than in-sample ones?

When doing linear regression the classic trap is to report residuals on the training data rather than testing data. However I seem to be consistently getting significantly lower errors on my test data ...
0
votes
0answers
24 views

Ways to describe matrix whose elements are linear combinations of elements of a vector?

The motivation for this question comes from trying to solve the following system: $$B = AX: \left[X\right]_{ij}=\sum_{l=1}^k \alpha_{k}^{(ij)}c_k$$ Where $A,B \in \mathbb{R}^{N\times p}, \hspace{0.1in}...
0
votes
0answers
6 views

What is random design in local constant estimation?

Suppose we have paired observations $\left(x_{i}, Y_{i}\right)$ and assume the nonparametric model $$ Y_{i}=f\left(x_{i}\right)+\epsilon_{i}, \quad i=1, \ldots, n $$ where $Y_{i}$ 's are responses, ...
0
votes
0answers
6 views

What is a relationship between standard deviation of a dependent variable and standard deviation of residuals?

In a book, ThinkStats, by Allen Downey, it uses an example of a simple linear regression model to predict babies'' birth weights using mothers' ages. Then, it mentions that standard deviation of the ...
0
votes
0answers
12 views

Is the formula for standard error for the slope of a linear regression with intercept the same as without?

If we are given sets $X$ and $Y$. The standart error formula for $\alpha$ coefficient of the regrssion $\hat{y} = \alpha x + \beta$ is $$ \frac{\sum{(y_i -\hat{y})^2}/(n-2)}{\sqrt{\sum(x_i-\bar{x})^2}}...
0
votes
0answers
18 views

Does the equality coefficients of linear regression of X onto Y and Y onto X imply coincidence of the lines?

Let's assume that we consider the model without an intercept $\hat{y_i} = x_i\hat{\beta}$. So, the formula for $\hat{\beta}$ is $\frac{ \sum\limits_{i=1}^n x_i y_i}{\sum\limits_{j=1}^n x_j^2}$. I ...
0
votes
0answers
8 views

Solving Matrix Equation in Residual Sum of Squares when Inputs not Linearly Independent

I'm reading Elements of Statistical Learning. On page 46, it talks about the residual sum of squares. We want to minimize the residual sum of squares. Let $X$ be a $N \times p+1$ matrix. Let $\beta$ ...
0
votes
1answer
22 views

Are $\beta_0$ and $\beta_1$ unbiased estimators of $\hat\beta_0$ and $\hat\beta_1$?

When we are discussing simple linear regression with: $$Y_i = \beta_0 + X_i\beta_1 +u_i$$ $\hat\beta_0$ and $\hat\beta_1$ are estimates of this model using OLS. With a simple proof we get $E(\hat\...
3
votes
1answer
29 views

Does quadratic risk of MLE for multivariate linear regression go to zero with more and more data?

For the simple multivariate linear regression with Gaussian noise: $\mathbf{Y} = \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\epsilon}$, where $\mathbf{Y} \in \mathbb{R}^n$: the vector of dependent ...
0
votes
0answers
15 views

Most Important Variables Linear Regression

When determining the most important variables in linear regression, I understand that first we need to decrease multicollinearity, if present, by using some sort of variable reduction method such as ...
0
votes
0answers
21 views

Confidence Interval of linear regression model - Standard Deviation vs Standard Error

For a (large) experimental dataset I am assuming a quadratic model and perform a regression analysis. Then I'd like to have some information about the accuracy of my model parameters, so I'm ...
0
votes
0answers
8 views

Derivation of multiple linear regression model, the 'matrix' problem.

I'm studying multiple linear regression. There're so many posts about the derivation of formula. But I can't find the one fully explaining how to deal with the matrix. I'm not good at linear algebra ...
2
votes
2answers
99 views

Regression linear model without error term

I have this linear model from a regression: $Y_i$ = $\beta_1X_{i1}$ + ... + $\beta_mX_{im}$ + $\epsilon_i$ The matrix representation is: $Y$ = $X\beta$ + $\epsilon$ In a lot of places like Wikipedia ...
1
vote
0answers
71 views

Covariance between estimated y and observed y(in simple linear regression)

I wonder what is $Cov(y,\hat{y})$ the Covariance between estimated $\hat{y}$ and observed value $y$, in the situation of simple linear regression. Some Youtube tutor said it's 'O', but i can't agree ...
0
votes
0answers
11 views

How to decode/understand the math behind ACF and PACF?

For the past month I have been trying to understand the math behind the autocorrelation function and partial autocorrelation function for time-series project I have been working on. However, I am only ...
1
vote
0answers
48 views

Continuity between Polynomial Fits

I'm currently working on a polynomial regression project and I've hit a wall. I'm trying to fit time-series data where I wish to "scroll" a window across the data and fit a polynomial to the ...
0
votes
0answers
37 views

Gradient descent's cost function: Mean Squared Error vs. Sum of Squared Errors

In many introductory Machine Learning textbooks or online resources, the cost function to be optimized with gradient descent to find a linear regression model is the Mean Squared Error (MSE), defined ...
0
votes
1answer
42 views

Distribution of adjusted R squared under null

Assume the standard linear regression model \begin{equation} Y = \beta_0 + \beta_1 X_1 + \cdots + \beta_k X_k + \epsilon \end{equation} Under $H_0: \beta_1 = \beta_2 = \cdots = \beta_k = 0$, we know $...
0
votes
0answers
23 views

SVD of a matrix storing gradients of a least squares regression problem

Suppose we have $n$ linear and noisy measurements of the form $y_i = x_i^\top\beta^\ast+e_i$ from an unknown parameter $\beta^\ast\in \mathbb{R}^d$. For each measurement consider the $\ell_2$ loss $...
0
votes
1answer
19 views

Conditions for $\beta_0=\bar{y}$ in simple linear regression using least squares

If we consider a linear model $\mathbf{y}=\beta_0\mathbf{1}+\beta_1\mathbf{a_1}+\beta_0\mathbf{a_2}+\varepsilon$ with $a_1,a_2$ column vectors with $n$ entries and $\beta_1\neq\beta_2\neq 0$. Then, ...
0
votes
0answers
15 views

What are diagonal weighted matrices in Total Least-Square Approach

I am going through Total least square and came across two diagonal matrices $D$ and $T$ $$ D= \operatorname{diag}(d_1, ...,d_m)$$ $$ T = \operatorname{diag}(t_1,...,t_{n+k})$$ Total Least Square $$(A+ ...
0
votes
1answer
23 views

Why does $\frac{\delta}{\delta\beta}y^TX\beta=\frac{\delta}{\delta\beta}B^TX^Ty?$

Why does $\frac{\delta}{\delta\beta}y^TX\beta=\frac{\delta}{\delta\beta}B^TX^Ty?$ In linear regression the parameters to the function $y=X\beta + \epsilon$ can be found by calculating the derivative ...
0
votes
1answer
29 views

Proof that expected value of estimators for Ordinary Least Squares estimators equals the optimal solution?

I'm doing a linear regression master's course right now and the prof wrote "Properties of OLS estimators is that they are unbiased estimators: $$\mathbb{E}(\hat{B})=\mathbb{E}[(X^TX)^{-1}X^Ty]$$ $...
0
votes
0answers
17 views

Show that the model with highest $AIC$ is the model with the lowest Mallows $C_p$ statistic.

$AIC = l_S - |S|, C_p = \hat{R_{tr}(S)} + 2|S|\hat{\sigma}^2$, $|S|$-is the number of the columns in design matrix, $\hat{R_{tr}(S)} = \sum_{i=1}^{n}(y-\hat{y}(S))^2$. Assume a linear regression model ...
1
vote
1answer
33 views

Transformation of function confusion

I am a bit confused about a concept that was recently covered in my stats class. My professor said that there's never any real need to do nonlinear regression, because a function can always be ...
1
vote
2answers
34 views

Ridge regression estimator in high-dimensions: is $(X^TX + \epsilon I_p)^{-1}X^Ty$ finite as $\epsilon \rightarrow 0$?

Consider the ridge regression estimator $$\hat{\beta}_{\epsilon} := (X^TX + \epsilon I_p)^{-1}X^Ty$$ where $X$ is an $n$ by $p$ matrix with $n < p$. Let $\| \hat{\beta}_{\epsilon} \|_{1} := \sum_{j=...
0
votes
1answer
21 views

Can we estimate $ y = \beta_1\exp(x) + \beta_2\exp(-x) + \beta_3$ using Linear Regression

Can we estimate the following relationship using linear regression. Here, $\beta_1, \beta_2 $ and $\beta_3$ are parameters. $$ y = \beta_1\exp(x) + \beta_2\exp(-x) + \beta_3$$
0
votes
0answers
29 views

Intuition about a matrix multiplication equality

I encountered the following formula while studying the analytical solution of the Linear regression problem in Machine learning context (Optimizing the weight w.r.t squared error) $w^{T}X^{T}y=y^{T}Xw$...
-2
votes
1answer
86 views

Coefficient of Determination and Standard Error of the Model

Background explaining standard concepts and standard terminology used in linear regression and analysis of variance: It will be supposed that one has data points $(X_i,Y_i),\, i = 1,\ldots,n.$ The ...
0
votes
0answers
14 views

Closed form solution for Restricted Weighted Least Squares

From Greene, we know that the closed-form solution of a restricted least squares is: $\beta_{Constrained} = \beta_{Uncon} - (X'X)^{-1}R'[R(X'X)^{-1}R']^{-1}(R\beta_{Uncon}-r)$. Is there any similar ...
0
votes
0answers
22 views

Is there an analytical solution to MLE of linear regression with non-normal(exponential) error term?

I know that MLE of normal linear regression $y = k_1x_1+k_2x_2 + \epsilon, \epsilon\sim N(0,1)$ has a nice analytical solution. But what if the error term is exponential distribution? the error term ...
5
votes
5answers
180 views

Does $(X'X)^{-1}$ always exist?

I'm studing Machine Learning theory and I have a questions about Normal Equation. Normal Equation is: $\Theta = (X'X)^{-1}X'Y\tag 1$ I now that ( in some cases) we can use this other equation: $\Theta ...
0
votes
1answer
12 views

Matrix algebra in least squares regression

Consider this formula for least-squares regression: \begin{aligned}L(D,{\vec {\beta }})&=||X{\vec {\beta }}-Y||^{2}\\&=(X{\vec {\beta }}-Y)^{T}(X{\vec {\beta }}-Y)\\&=\color{red}{Y^{T}Y-Y^{...
-2
votes
1answer
63 views

Simple Linear Regression Machine Learning Course [closed]

Your friend in the U.S. gives you a simple regression fit for predicting house prices from square feet. The estimated intercept is -44850 and the estimated slope is 280.76. You believe that your ...
0
votes
1answer
44 views

Find rank of matrix [closed]

If $X$ is a $N \times D$ matrix with $(D\gg N)$ with $\operatorname{rank}(X) = N$, what is $\operatorname{rank}(X^T \cdot X)$ where $X^T$ is the transpose matrix of $X$? I am little new to linear ...
1
vote
1answer
33 views

Soft-EM: E-step for fitting mixed linear regression model

I want to derive the formulas for the soft EM algorithm for the following model $P[y_i | x_i, \pi_{1,\dots,m}, a_{1,\dots,m}] = \sum_{j=1}^m \pi_j \frac{1}{\sqrt{2\pi}\sigma} exp(-\frac{(a_j^T x_i - ...
2
votes
0answers
31 views

Linear regression with non-fixed regressors and some properties

I was talking to my teacher the other day about the OLS and linear regression model $Y = \beta X + \varepsilon$. If the regressors X are fixed numbers and I don't have the normal condition on the ...
1
vote
0answers
26 views

Consistency of least squares for linear regression

Consider the generative model for linear regression w.r.t. the true parameter $w^* \in S^{d-1}$ $$y=Xw^*+e$$ with i.i.d. Gaussian error $e \sim N(0, \sigma^2I_n)$. Let $X \in \mathbb{R}^{n\times d}$ ...
0
votes
1answer
20 views

Coefficient Estimators of $\frac{1}{x^{2}}$ Weighted Least Squares Linear Regression

I have a feeling there should be a mathematical formular for determining the estimators of the coefficients of a $\frac{1}{x^{2}}$ Weighted Linear Regression. I was able to derive the estimators ($a$ ...
1
vote
1answer
20 views

How can we interpret residual plot in case we have many variables?

In Residual plots, we try to visualize & interpret whether linearity is valid or not in the linear regression model. One way to do this is to plot error term wrt to the independent variable(say x)....
0
votes
0answers
30 views

What do the Eigenvalues and Eigenvectors Of A Coefficient Matrix Tell You?

I see eigenvalues and eigenvectors talked about a lot in relation to a coefficient matrix (in regression and time series models). What is the significance of eigenvalues and eigenvectors in this ...
1
vote
0answers
34 views

Confidence Interval for E(Y0) - Linear Regression

I'm solving exercises as preparation for a statistics exam and I'm scratching my head on the following exercise. I'll start by giving the data I have to work with: n = 10; x (average) = 14; y (average)...

1
2 3 4 5
18