Questions tagged [linear-regression]
For questions about linear regressions, an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables.
1,136
questions
1
vote
0
answers
17
views
Why does the Lasso only select only $n$ variables if $p \gg n$
$L_1$ or lasso regularization in regression problems is defined as $$\min||X\beta - y||_2^2 + \lambda ||\beta||_1$$ Multiple resources point out that for $X\in \mathbb{C}^{n\times p}$ if $p \gg n$ ...
- 117
0
votes
0
answers
8
views
Which $n$ values does lasso pick if $p \gg n$
It's known that lasso regression picks only $n$ variables in a matrix $X^{n\times p}$ matrix where $p \gg n$ but which variables exactly does lasso pick?
Intuitively lasso should always pick the ...
- 117
1
vote
1
answer
48
views
efficiency of estimator, $\overline{\frac{1}{X^2}}$ vs $\frac{1}{\overline{X}^2}$ vs $\frac{1}{\overline{X^2}}$
I was studying point estimator, and I tried to compare the variances of the estimators to find out which one is more efficient. (Hogg, Tanis "Probability and Statistical Inference" Ch.6)
It ...
- 11
1
vote
1
answer
51
views
Why does least-squares need regularization?
If I understand regularization correctly, it helps if a least-squares problem is not well-posed thus...
the problem has no solution
the problem has multiple solutions
a small change in the input ...
- 117
1
vote
0
answers
29
views
preventing rare extreme values in linear regression prediction
I am trying to train a model with a lot of input variables using linear regression. For technical reasons, my training data is obtained from a simulation that closely but not perfectly mirrors the ...
-1
votes
0
answers
18
views
Puzzled in Parametric Linear Regression Matrix Solution [closed]
If Ynx1 = Xnxpθpx1, then θ = (XTX)-1XTY.
But this doesn't work IRL (e.g. X = [1,2,3]T, Y = [2,4,5]T). I cannot think of why (it should be obvious though).
Any help on this part? Thank you!
Edit: Also, ...
0
votes
1
answer
17
views
If there is a high correlation between two variables, should we expect the high linear regression coefficient? [closed]
I have a data set with multiple features, let suppose $x_1,x_2,x_3,x_4$ and my dependent variable is $y$, when I compute the correlation matrix for $y,x_1,x_2,x_3,x_4$ then imagine the correlation ...
- 1
0
votes
1
answer
30
views
How to show that $\sum_{i=1}^n\left(x_i^2 - \bar{x}^2\right) = \sum_{i=1}^n\left(x_i - \bar{x}\right)^2$
I am reading through a book on linear regression and I am confused as to how a derivation has been done. The derivation up to where I have got is below.
$$ \sum_{i=1}^nx_i^2 - \frac{\left(\sum_{i=1}^...
- 119
0
votes
0
answers
14
views
Expectation of the Estimated Slope
In the following book, the expectation of the estimated slope with linear regression is derived as follows:
$$
\begin{align*}
\hat{\beta}_1 ~ &= ~ \frac{\frac{1}{n}\sum_{i=1}^n (x_i - \bar{x})(Y_i ...
- 167
1
vote
0
answers
12
views
Interpreting model accuracy as proof of real association in Machine Learning
Let $\mathbb{X}, \mathbb{Y}$ be training and test sets respectively for some data we assume comes from a function$f$. Let $\hat{f}(\theta)$ be a model of $f$ with a parameter vector $\theta$. Assume ...
- 1,356
-1
votes
0
answers
16
views
How to add two points to a set of points such that the sign of slope of the linear regression fit doesn't change?
I have a set of points $\{(i, y_i)\}_{i=1}^k$, and fit a linear regression line on it, yielding $y_i = p_0 i + p_1$. Now I add two more points $(k+1, y_{k+1}), (k+2, y_{k+2})$ to the set of points, ...
- 1
0
votes
0
answers
8
views
OLS Beta estimate with indicator function
I have $Y=\alpha + \beta *1_{A}+\epsilon$, then can I just as normally compute
$\hat{\beta}=\frac{Cov(Y,1_{A})}{Var(1_{A})}$?
Where the event $A$ and $\epsilon$ are correlated.
Thanks!
- 55
0
votes
0
answers
9
views
MSE of WLS estimator with biased measurements
(also posted on CV, but I will try here too)
I am trying to find out if what I am looking at is a known problem.
I am considering the case of weighted least squares, and I am trying to find the ...
- 328
0
votes
0
answers
9
views
1
vote
0
answers
22
views
Bagging Linear Model?
I have a question regarding bagging linear models.
Suppose you wanna do linear regression on data X and y.
Alice directly implements (OLS) Linear Regression on it. The model is A1.
Bob applies bagging....
0
votes
0
answers
20
views
What do you call the value you get from a regression equation?
Say I have 2 sets of data which I perform a linear regression on, now I have a regression equation in the form y=mx+b (for example).
Now lets say when x = 1, the observed/real value of y = 1.
So now ...
- 1
0
votes
0
answers
11
views
Ordinary least squares and Gauss-Markov theorem
Let
$$Y=X\beta+\varepsilon,\quad Y,\varepsilon\in\mathbb{R}^n,\quad X\in\mathbb{R}^{n\times p},\quad \beta\in\mathbb{R}^p$$
be the standard linear regression model and $\widehat{\beta}$ the OLS ...
- 342
0
votes
0
answers
20
views
Minimizing the Objective Function for Derivation of Least Squares Estimator for Multiple Linear Regression
I have a Multiple Linear Regression model where the error vector $\underline{e} \sim N(\underline{0},\underline{R})$
$$\underline{z} = \underline{H}\,\underline{x}_{true} + \underline{e}$$
When I ...
- 43
0
votes
0
answers
11
views
Variance of predicted value in linear regression model
I am working on a question whereby I have calculated the linear regression model
$$Y_i = -20.57 + 1.7446x_i + \epsilon_i ,$$
and have been given the data that $x_i$ = 85. From this, I have calculated ...
- 23
0
votes
0
answers
13
views
Computing 95% Confidence interval for the slope of a simple linear regression out of Minitab output
I created a linear regression model and I have the following Minitab output. I am trying to compute the 95% confidence interval for the slope of the true regression line.
I know that the formula for ...
0
votes
0
answers
17
views
Does Multiple Linear Regression work with stocks? If so, does this make sense?
I have a dataset that contains information on a stock. In a csv file, prices are organized like
this
I am able to get the equation for the line, where y = adjusted close prediction, x = index ...
- 1
0
votes
1
answer
36
views
OLS coefficient when errors are not normally distributed
Assume that in this regression Y=β0+β1x+ϵ, where ϵ follows a Poisson distribution. Using OLS, estimate β0,β1 and cov(β1,β0).
I am wondering does the distribution of errors changes the β0 and β1 OLS ...
0
votes
0
answers
17
views
Bias in Reverse Regression with Measurement Error?
Lets say I want to estimate the following population regression: y = xb + v (v is the error term) and x is measured with error so that x = x + u. We know that my estimate of x is know biased by ...
- 3
0
votes
0
answers
10
views
Dependent errors leading to artificially small p values?
Why do dependent regression errors in a Multiple Linear Regression model (a violation of the assumptions of the MLR model) lead to underestimated standard errors and artificially small p-values? What ...
- 1,732
0
votes
0
answers
20
views
Interpreting a linear regression model
The relationship between annual average temperature over 10 years in various towns and the area of the UK in which the town is located was investigated. Area is described as being one of five ordered ...
1
vote
1
answer
20
views
Prove that $\text{rank}(\mathbf{I}-\mathbf{X}(\mathbf{X}^\intercal\mathbf{X})^{-1}\mathbf{X}^\intercal)=n-k-1$, where $\mathbf{X}$ is $n\times k+1$
This is required to show that $\text{SSR}/\sigma^2$ is $\chi^2(n-k-1)$, where SSR is the residual sum of squares $(\mathbf{y}-\mathbf{X}\boldsymbol{\hat{\beta}})^\intercal(\mathbf{y}-\mathbf{X}\...
- 157
1
vote
1
answer
47
views
MGF of quadratic form $\mathbf{y}^\intercal \mathbf{A} \mathbf{y}$ where $y\sim N_p(\mathbf{\mu},\mathbf{\Sigma})$
Theorem 5.2b of Linear Models in Statistics by Rencher and Schaalje is
If $\mathbf{y}$ is $N_p(\mathbf{\mu},\mathbf{\Sigma})$, then the moment generating function of $\mathbf{y}^\intercal\mathbf{A}\...
- 157
1
vote
0
answers
14
views
How is ANOVA different from a linear model?
This may be a naive question, but I realize all statistical tests of significance are inherently related to linear models and for anova in particular: I don't exactly understand the difference between ...
- 283
2
votes
0
answers
9
views
Is it possible that the correlation between $\hat{b}$ and $\hat{c}$ can be negative multiple linear regression?
Given the following linear regression model as following, with two explanatory variables $x_1$ and $x_2$ and response $y$
$$y_i=a+bx_{i1}+cx_{i2}+\epsilon_{i}$$
We say that $\hat{a}, \hat{b}, \hat{c}$ ...
- 405
0
votes
1
answer
38
views
Least squares estimation for linear regression model with random design
Suppose we have a model
$$y_i = g(x_i) + \epsilon_i, \quad i = 1, \ldots, n$$
where $y_i \in \mathbb{R}$ is a response variable, $x_i \in \mathcal{X} \subseteq \mathbb{R}^d$, and $\epsilon_i$ is the ...
- 1,940
0
votes
0
answers
19
views
Design matrix of a model
Suppose we have a model $y_{i} = \beta_{0} + \beta_{1} x_{i1} + \beta_{2}x_{i2} + \epsilon_{i}$. Then I need to find a general expression for $\hat{\sigma}²(X^TX)^{-1}$.
My first thought was that we ...
- 399
1
vote
0
answers
32
views
How would the linear regression model look like?
Typically I see $y$ as the response variable and $x$ as the predictor variable.
$y_i=\beta_0+\beta_1x_i+\epsilon_i,\quad i=1,\dots,n.$$
What other forms can the equation have? Would it be valid if we ...
- 31
0
votes
1
answer
38
views
Simple linear regression (sum of residuals and predictor)
Show explicitly that the following identity holds under a Simple Linear Regression:
$$
\ \sum_{i=1}^n r_i \hat{\mu_i} =0$$
with residuals $ r_i = y_i − \hat{\mu_i} $ and $\hat{\mu_i} = \hat{\beta_0}+\...
- 77
0
votes
0
answers
12
views
How to determine an autoregressive model with external inputs
I have data vectors $\mathbf{p}, \mathbf{q}$, and $\mathbf{r}$. There should be an approximate causal relationship between the data:
$ \mathbf{p}(t+1) = \alpha_1 \mathbf{p}(t) + \alpha_2 \mathbf{p}(t-...
- 75
1
vote
0
answers
25
views
Reducing variance in linear regression
While reading The Elements of Statistical Learning the author states that by shriking the coefficients of a liinear regression you raise the bias while lowering the variance and thus, sometimes, ...
0
votes
0
answers
21
views
Is it possible to use a chunk of observations instead of one observation in Recursive Least Squares (RLS) at once?
Recursive Least Squares (RLS) by its structure reestimates coefficients iteratively utilizing one new observation in each iteration. Is it possible to use $n$ new observations in one iteration to ...
0
votes
0
answers
13
views
linear fit to 2D wrapped phase
Suppose we have a number of noisy space-phase data, i.e., $(x_i,y_i,\phi_i)$, and we know they are subject to a 2D linear phase pattern.
That is to say, these data can be fitted using a 3-parameter ...
- 93
0
votes
0
answers
24
views
How to fit a ray into a set of 2D points, having the first point as pivot?
I am trying to improve the line simplification algorithm from Ramer-Douglas-Peucker by adding some kind of line fitting into it.
Today, this algorithm does not provide the simplification with ...
- 113
0
votes
0
answers
18
views
How to calculate multiple linear regression with more than 2 variable by hand
I found that most of the Multiple Linear Regression only 2 variable.
How to calculate 3 or more variable by hand?
- 117
0
votes
1
answer
40
views
Simple linear regression model with intercept parameter
Consider a Simple Linear Regression model with intercept parameter $\beta_1$ included. Show that the following holds:
$\sum\limits_{i=1}^n (y_i-\hat\mu_i)^2 = \sum\limits_{i=1}^n (y_i-\bar{y})^2 - \...
- 77
0
votes
0
answers
116
views
Prove that SSR/$\sigma^2$ follows a non-central chi-square distribution with degree of freedom p
We have the following theorem:
If $\mathbf{Z} \sim N(\mu, \mathbf{V}\sigma^2)$ for a singular matrix $\mathbf{V}$ and $\mathbf{AV}$ is idempotent then a quadratic form $\mathbf{Z}'(\mathbf{A}/\sigma^2)...
- 21
0
votes
0
answers
22
views
Regression, can someone help me understand this equation?
The criterion for mapping the individual test case to the four ideal functions is that the existing maximum
deviation of the calculated regression does not exceed the largest deviation between ...
0
votes
2
answers
44
views
Is the sum of residuals times a regressor equal to zero?
Suppose I am approximating the following system using ordinary least squares regression
$$y = p_0 + p_1 x_1 + p_2 x_2 + ... p_M x_M = \xi P$$
I know that a property of the least squares estimator is ...
- 11
0
votes
0
answers
22
views
High-dimensional generalization of Penrose best approximate solution to least square equation.
Fix $A\in\mathbb{R}^{d\times d}$. The solution to
$$ \min_{x\in\mathbb{R}^d: Ax=b}\Vert x \Vert_2^2 $$
is $A^{\dagger} b$. Roger Penrose showed (Corollary 1) that the the solution to
$$ \min_{X\in\...
- 33
1
vote
0
answers
31
views
Computing MLE with a linear regression model
I am stuck on how to compute the MLE for $β$ given the model $y_i = βx_i + \varepsilon_i$. I know that $β$ is an unknown slope, the $x_i$ are deterministic inputs, and the $y_i$ are random
...
- 21
1
vote
0
answers
69
views
what is the intuition behind the dual hat matrix being the identity matrix?
Given some data matrix $\mathbf{X} \in \mathbb{R}^{N\times n}$ the $\textit{hat matrix}$ is $\mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T$ (henceforth called $\mathbf{H}_{Primal}(\...
- 73
0
votes
0
answers
20
views
Linear regression with random coefficients
I am facing a problem defined by the following equation
$$\overrightarrow{x}+\overrightarrow{\epsilon}=\overrightarrow{c}$$
Where:
All the vectors contain $n$ entries,
$\overrightarrow{c}$ is a ...
- 159
0
votes
0
answers
28
views
A question about Mallows Cp Statistics
In the book "Introduction to Multiple Linear Regression" By Peck at.el., Mallows $C_p$ value is discussed in the Chapter 10. $C_p$ is defined as
$$C_p = \frac{SS_{\rm Res}(p)}{\widehat{\...
- 11
0
votes
0
answers
26
views
Confidence interval of slope parameter when the intercept in known in simple linear regression model
It is one of the exercise in "Introduction to Multiple Linear Regression".
Consider the usual linear regression model
\begin{equation}
y = \beta_0+ \beta_1 x +\epsilon
\end{equation}
where $\...
- 11
0
votes
1
answer
32
views
Econometrics/Statistics Regression Question [closed]
As you can see from the provided picture given Heart attack given rate per 100,000 population. I was able to successfully ran my regression; but now I am trying to figure out how to alter my ...
- 49