Questions tagged [linear-regression]

For questions about linear regressions, an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables.

Filter by
Sorted by
Tagged with
1 vote
0 answers
17 views

Why does the Lasso only select only $n$ variables if $p \gg n$

$L_1$ or lasso regularization in regression problems is defined as $$\min||X\beta - y||_2^2 + \lambda ||\beta||_1$$ Multiple resources point out that for $X\in \mathbb{C}^{n\times p}$ if $p \gg n$ ...
user avatar
0 votes
0 answers
8 views

Which $n$ values does lasso pick if $p \gg n$

It's known that lasso regression picks only $n$ variables in a matrix $X^{n\times p}$ matrix where $p \gg n$ but which variables exactly does lasso pick? Intuitively lasso should always pick the ...
user avatar
1 vote
1 answer
48 views

efficiency of estimator, $\overline{\frac{1}{X^2}}$ vs $\frac{1}{\overline{X}^2}$ vs $\frac{1}{\overline{X^2}}$

I was studying point estimator, and I tried to compare the variances of the estimators to find out which one is more efficient. (Hogg, Tanis "Probability and Statistical Inference" Ch.6) It ...
user avatar
1 vote
1 answer
51 views

Why does least-squares need regularization?

If I understand regularization correctly, it helps if a least-squares problem is not well-posed thus... the problem has no solution the problem has multiple solutions a small change in the input ...
user avatar
1 vote
0 answers
29 views

preventing rare extreme values in linear regression prediction

I am trying to train a model with a lot of input variables using linear regression. For technical reasons, my training data is obtained from a simulation that closely but not perfectly mirrors the ...
user avatar
-1 votes
0 answers
18 views

Puzzled in Parametric Linear Regression Matrix Solution [closed]

If Ynx1 = Xnxpθpx1, then θ = (XTX)-1XTY. But this doesn't work IRL (e.g. X = [1,2,3]T, Y = [2,4,5]T). I cannot think of why (it should be obvious though). Any help on this part? Thank you! Edit: Also, ...
user avatar
0 votes
1 answer
17 views

If there is a high correlation between two variables, should we expect the high linear regression coefficient? [closed]

I have a data set with multiple features, let suppose $x_1,x_2,x_3,x_4$ and my dependent variable is $y$, when I compute the correlation matrix for $y,x_1,x_2,x_3,x_4$ then imagine the correlation ...
user avatar
  • 1
0 votes
1 answer
30 views

How to show that $\sum_{i=1}^n\left(x_i^2 - \bar{x}^2\right) = \sum_{i=1}^n\left(x_i - \bar{x}\right)^2$

I am reading through a book on linear regression and I am confused as to how a derivation has been done. The derivation up to where I have got is below. $$ \sum_{i=1}^nx_i^2 - \frac{\left(\sum_{i=1}^...
user avatar
0 votes
0 answers
14 views

Expectation of the Estimated Slope

In the following book, the expectation of the estimated slope with linear regression is derived as follows: $$ \begin{align*} \hat{\beta}_1 ~ &= ~ \frac{\frac{1}{n}\sum_{i=1}^n (x_i - \bar{x})(Y_i ...
user avatar
1 vote
0 answers
12 views

Interpreting model accuracy as proof of real association in Machine Learning

Let $\mathbb{X}, \mathbb{Y}$ be training and test sets respectively for some data we assume comes from a function$f$. Let $\hat{f}(\theta)$ be a model of $f$ with a parameter vector $\theta$. Assume ...
user avatar
  • 1,356
-1 votes
0 answers
16 views

How to add two points to a set of points such that the sign of slope of the linear regression fit doesn't change?

I have a set of points $\{(i, y_i)\}_{i=1}^k$, and fit a linear regression line on it, yielding $y_i = p_0 i + p_1$. Now I add two more points $(k+1, y_{k+1}), (k+2, y_{k+2})$ to the set of points, ...
user avatar
  • 1
0 votes
0 answers
8 views

OLS Beta estimate with indicator function

I have $Y=\alpha + \beta *1_{A}+\epsilon$, then can I just as normally compute $\hat{\beta}=\frac{Cov(Y,1_{A})}{Var(1_{A})}$? Where the event $A$ and $\epsilon$ are correlated. Thanks!
user avatar
  • 55
0 votes
0 answers
9 views

MSE of WLS estimator with biased measurements

(also posted on CV, but I will try here too) I am trying to find out if what I am looking at is a known problem. I am considering the case of weighted least squares, and I am trying to find the ...
user avatar
  • 328
0 votes
0 answers
9 views

Linearregression of two dataframes

I have two dataframes: ...
user avatar
  • 1
1 vote
0 answers
22 views

Bagging Linear Model?

I have a question regarding bagging linear models. Suppose you wanna do linear regression on data X and y. Alice directly implements (OLS) Linear Regression on it. The model is A1. Bob applies bagging....
user avatar
0 votes
0 answers
20 views

What do you call the value you get from a regression equation?

Say I have 2 sets of data which I perform a linear regression on, now I have a regression equation in the form y=mx+b (for example). Now lets say when x = 1, the observed/real value of y = 1. So now ...
user avatar
0 votes
0 answers
11 views

Ordinary least squares and Gauss-Markov theorem

Let $$Y=X\beta+\varepsilon,\quad Y,\varepsilon\in\mathbb{R}^n,\quad X\in\mathbb{R}^{n\times p},\quad \beta\in\mathbb{R}^p$$ be the standard linear regression model and $\widehat{\beta}$ the OLS ...
user avatar
  • 342
0 votes
0 answers
20 views

Minimizing the Objective Function for Derivation of Least Squares Estimator for Multiple Linear Regression

I have a Multiple Linear Regression model where the error vector $\underline{e} \sim N(\underline{0},\underline{R})$ $$\underline{z} = \underline{H}\,\underline{x}_{true} + \underline{e}$$ When I ...
user avatar
0 votes
0 answers
11 views

Variance of predicted value in linear regression model

I am working on a question whereby I have calculated the linear regression model $$Y_i = -20.57 + 1.7446x_i + \epsilon_i ,$$ and have been given the data that $x_i$ = 85. From this, I have calculated ...
user avatar
0 votes
0 answers
13 views

Computing 95% Confidence interval for the slope of a simple linear regression out of Minitab output

I created a linear regression model and I have the following Minitab output. I am trying to compute the 95% confidence interval for the slope of the true regression line. I know that the formula for ...
user avatar
0 votes
0 answers
17 views

Does Multiple Linear Regression work with stocks? If so, does this make sense?

I have a dataset that contains information on a stock. In a csv file, prices are organized like this I am able to get the equation for the line, where y = adjusted close prediction, x = index ...
user avatar
  • 1
0 votes
1 answer
36 views

OLS coefficient when errors are not normally distributed

Assume that in this regression Y=β0+β1x+ϵ, where ϵ follows a Poisson distribution. Using OLS, estimate β0,β1 and cov(β1,β0). I am wondering does the distribution of errors changes the β0 and β1 OLS ...
user avatar
0 votes
0 answers
17 views

Bias in Reverse Regression with Measurement Error?

Lets say I want to estimate the following population regression: y = xb + v (v is the error term) and x is measured with error so that x = x + u. We know that my estimate of x is know biased by ...
user avatar
  • 3
0 votes
0 answers
10 views

Dependent errors leading to artificially small p values?

Why do dependent regression errors in a Multiple Linear Regression model (a violation of the assumptions of the MLR model) lead to underestimated standard errors and artificially small p-values? What ...
user avatar
  • 1,732
0 votes
0 answers
20 views

Interpreting a linear regression model

The relationship between annual average temperature over 10 years in various towns and the area of the UK in which the town is located was investigated. Area is described as being one of five ordered ...
user avatar
1 vote
1 answer
20 views

Prove that $\text{rank}(\mathbf{I}-\mathbf{X}(\mathbf{X}^\intercal\mathbf{X})^{-1}\mathbf{X}^\intercal)=n-k-1$, where $\mathbf{X}$ is $n\times k+1$

This is required to show that $\text{SSR}/\sigma^2$ is $\chi^2(n-k-1)$, where SSR is the residual sum of squares $(\mathbf{y}-\mathbf{X}\boldsymbol{\hat{\beta}})^\intercal(\mathbf{y}-\mathbf{X}\...
user avatar
  • 157
1 vote
1 answer
47 views

MGF of quadratic form $\mathbf{y}^\intercal \mathbf{A} \mathbf{y}$ where $y\sim N_p(\mathbf{\mu},\mathbf{\Sigma})$

Theorem 5.2b of Linear Models in Statistics by Rencher and Schaalje is If $\mathbf{y}$ is $N_p(\mathbf{\mu},\mathbf{\Sigma})$, then the moment generating function of $\mathbf{y}^\intercal\mathbf{A}\...
user avatar
  • 157
1 vote
0 answers
14 views

How is ANOVA different from a linear model?

This may be a naive question, but I realize all statistical tests of significance are inherently related to linear models and for anova in particular: I don't exactly understand the difference between ...
user avatar
  • 283
2 votes
0 answers
9 views

Is it possible that the correlation between $\hat{b}$ and $\hat{c}$ can be negative multiple linear regression?

Given the following linear regression model as following, with two explanatory variables $x_1$ and $x_2$ and response $y$ $$y_i=a+bx_{i1}+cx_{i2}+\epsilon_{i}$$ We say that $\hat{a}, \hat{b}, \hat{c}$ ...
user avatar
0 votes
1 answer
38 views

Least squares estimation for linear regression model with random design

Suppose we have a model $$y_i = g(x_i) + \epsilon_i, \quad i = 1, \ldots, n$$ where $y_i \in \mathbb{R}$ is a response variable, $x_i \in \mathcal{X} \subseteq \mathbb{R}^d$, and $\epsilon_i$ is the ...
user avatar
  • 1,940
0 votes
0 answers
19 views

Design matrix of a model

Suppose we have a model $y_{i} = \beta_{0} + \beta_{1} x_{i1} + \beta_{2}x_{i2} + \epsilon_{i}$. Then I need to find a general expression for $\hat{\sigma}²(X^TX)^{-1}$. My first thought was that we ...
user avatar
  • 399
1 vote
0 answers
32 views

How would the linear regression model look like?

Typically I see $y$ as the response variable and $x$ as the predictor variable. $y_i=\beta_0+\beta_1x_i+\epsilon_i,\quad i=1,\dots,n.$$ What other forms can the equation have? Would it be valid if we ...
user avatar
  • 31
0 votes
1 answer
38 views

Simple linear regression (sum of residuals and predictor)

Show explicitly that the following identity holds under a Simple Linear Regression: $$ \ \sum_{i=1}^n r_i \hat{\mu_i} =0$$ with residuals $ r_i = y_i − \hat{\mu_i} $ and $\hat{\mu_i} = \hat{\beta_0}+\...
user avatar
  • 77
0 votes
0 answers
12 views

How to determine an autoregressive model with external inputs

I have data vectors $\mathbf{p}, \mathbf{q}$, and $\mathbf{r}$. There should be an approximate causal relationship between the data: $ \mathbf{p}(t+1) = \alpha_1 \mathbf{p}(t) + \alpha_2 \mathbf{p}(t-...
user avatar
1 vote
0 answers
25 views

Reducing variance in linear regression

While reading The Elements of Statistical Learning the author states that by shriking the coefficients of a liinear regression you raise the bias while lowering the variance and thus, sometimes, ...
user avatar
0 votes
0 answers
21 views

Is it possible to use a chunk of observations instead of one observation in Recursive Least Squares (RLS) at once?

Recursive Least Squares (RLS) by its structure reestimates coefficients iteratively utilizing one new observation in each iteration. Is it possible to use $n$ new observations in one iteration to ...
user avatar
0 votes
0 answers
13 views

linear fit to 2D wrapped phase

Suppose we have a number of noisy space-phase data, i.e., $(x_i,y_i,\phi_i)$, and we know they are subject to a 2D linear phase pattern. That is to say, these data can be fitted using a 3-parameter ...
user avatar
0 votes
0 answers
24 views

How to fit a ray into a set of 2D points, having the first point as pivot?

I am trying to improve the line simplification algorithm from Ramer-Douglas-Peucker by adding some kind of line fitting into it. Today, this algorithm does not provide the simplification with ...
user avatar
  • 113
0 votes
0 answers
18 views

How to calculate multiple linear regression with more than 2 variable by hand

I found that most of the Multiple Linear Regression only 2 variable. How to calculate 3 or more variable by hand?
user avatar
  • 117
0 votes
1 answer
40 views

Simple linear regression model with intercept parameter

Consider a Simple Linear Regression model with intercept parameter $\beta_1$ included. Show that the following holds: $\sum\limits_{i=1}^n (y_i-\hat\mu_i)^2 = \sum\limits_{i=1}^n (y_i-\bar{y})^2 - \...
user avatar
  • 77
0 votes
0 answers
116 views

Prove that SSR/$\sigma^2$ follows a non-central chi-square distribution with degree of freedom p

We have the following theorem: If $\mathbf{Z} \sim N(\mu, \mathbf{V}\sigma^2)$ for a singular matrix $\mathbf{V}$ and $\mathbf{AV}$ is idempotent then a quadratic form $\mathbf{Z}'(\mathbf{A}/\sigma^2)...
user avatar
  • 21
0 votes
0 answers
22 views

Regression, can someone help me understand this equation?

The criterion for mapping the individual test case to the four ideal functions is that the existing maximum deviation of the calculated regression does not exceed the largest deviation between ...
user avatar
0 votes
2 answers
44 views

Is the sum of residuals times a regressor equal to zero?

Suppose I am approximating the following system using ordinary least squares regression $$y = p_0 + p_1 x_1 + p_2 x_2 + ... p_M x_M = \xi P$$ I know that a property of the least squares estimator is ...
user avatar
  • 11
0 votes
0 answers
22 views

High-dimensional generalization of Penrose best approximate solution to least square equation.

Fix $A\in\mathbb{R}^{d\times d}$. The solution to $$ \min_{x\in\mathbb{R}^d: Ax=b}\Vert x \Vert_2^2 $$ is $A^{\dagger} b$. Roger Penrose showed (Corollary 1) that the the solution to $$ \min_{X\in\...
user avatar
  • 33
1 vote
0 answers
31 views

Computing MLE with a linear regression model

I am stuck on how to compute the MLE for $β$ given the model $y_i = βx_i + \varepsilon_i$. I know that $β$ is an unknown slope, the $x_i$ are deterministic inputs, and the $y_i$ are random ...
user avatar
  • 21
1 vote
0 answers
69 views

what is the intuition behind the dual hat matrix being the identity matrix?

Given some data matrix $\mathbf{X} \in \mathbb{R}^{N\times n}$ the $\textit{hat matrix}$ is $\mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T$ (henceforth called $\mathbf{H}_{Primal}(\...
user avatar
0 votes
0 answers
20 views

Linear regression with random coefficients

I am facing a problem defined by the following equation $$\overrightarrow{x}+\overrightarrow{\epsilon}=\overrightarrow{c}$$ Where: All the vectors contain $n$ entries, $\overrightarrow{c}$ is a ...
user avatar
  • 159
0 votes
0 answers
28 views

A question about Mallows Cp Statistics

In the book "Introduction to Multiple Linear Regression" By Peck at.el., Mallows $C_p$ value is discussed in the Chapter 10. $C_p$ is defined as $$C_p = \frac{SS_{\rm Res}(p)}{\widehat{\...
user avatar
0 votes
0 answers
26 views

Confidence interval of slope parameter when the intercept in known in simple linear regression model

It is one of the exercise in "Introduction to Multiple Linear Regression". Consider the usual linear regression model \begin{equation} y = \beta_0+ \beta_1 x +\epsilon \end{equation} where $\...
user avatar
0 votes
1 answer
32 views

Econometrics/Statistics Regression Question [closed]

As you can see from the provided picture given Heart attack given rate per 100,000 population. I was able to successfully ran my regression; but now I am trying to figure out how to alter my ...
user avatar
  • 49

1
2 3 4 5
23