1
$\begingroup$

When fitting a linear regression model in R for example, we get as an output all the coefficients $\beta_i$ along with some other properties like the standard deviation and a 95% CI for each coefficient.

For me, linear regression is an optimization problem, we're trying to find $\beta$ that minimizes : $$ \max_{i \,\in\, [|1,N|]} |y_i-\beta^Tx_i -\beta_0| $$ So hopefully we find and optimal $\hat{\beta}$. With this construction, I don't understand how this would result in any std for any $\beta_i$.

I guess programming languages like R and Python do this differently? Let's say we have a sample with $N$ $x_i$, do they actually perform the optimization on different subsets of the data, which would yield different optimums and then average them? I would really like to understand this aspect of linear regression because it's been bugging me for a while now.

Thank you all.

$\endgroup$
3
  • 1
    $\begingroup$ The optimization in ordinary regression (one predictor variable for simplicity) minimizes the sum of squared residuals $\sum_{i=1}^n(\hat \beta_0 + \hat \beta_1 x_i - Y_i)^2.$ The randomness comes from the $\sigma$ of the model $Y_i = \beta_0 + \beta_1 x_i + e_i,$ where $e_i$ are iid NORM(0, $\sigma$). So even the 'optimized' line may not be exactly correct. The parameter estimates $\hat \beta_0,\, \hat\beta_1,\,$ and $\hat \sigma$ are random variables, each with its own variance. $\endgroup$
    – BruceET
    Dec 12, 2017 at 0:19
  • 1
    $\begingroup$ Technically the estimators are the random variables and the estimates are outcomes of said random variables. $\endgroup$
    – CptB3RRY
    Dec 12, 2017 at 2:32
  • $\begingroup$ My answer initially had the wrong number of degrees of freedom for the chi-square random variable. It should have been $\chi^2_{n-p-1}.$ What I wrote would be correct only if $p-1.$ I've corrected it. $\endgroup$ Dec 18, 2017 at 19:05

1 Answer 1

0
$\begingroup$

You have observed values of $x_{1,i}, \ldots, x_{p,i}$ for $i=1,\dots,n$ and $n\gg p,$ and observed values of $y_i,$ and you want to estimate coefficients $\beta_0,\beta_1,\ldots,\beta_p$ for which

$$y_i = \beta_1+\beta_1 x_{1,i} + \cdots+ \beta_p x_{p,i} + \text{error}_i \text{ for } i = 1,\ldots, n. \tag 1 $$

The most usual method of estimation is that the estimates $\widehat \beta_0,\widehat\beta_1,\ldots,\widehat\beta_p$ are the values of $\beta_0,\beta_1,\ldots,\beta_p$ that minimize the sum of squares $$ \sum_{k=1}^n (y_i - (\beta_0 + \beta_1 x_{1,i} + \cdots + \beta_p x_{p,i}))^2. $$ That's how R does it if you use the lm command.

When it's done that way, the mapping $$ \left[ \begin{array}{c} y_1 \\ \vdots \\ \vdots \\ \vdots \\ y_n \end{array} \right] \mapsto \left[ \begin{array}{c} \widehat\beta_0 \\ \widehat\beta_1 \\ \vdots \\ \widehat\beta_p \end{array} \right] \qquad (\text{with all $x_{i,j}$ fixed}) $$ is linear. Call the vector to the left of the arrow $Y$ and the one to its right $\widehat\beta,$ and let $X$ be the $(p+1)\times n$ matrix $$ \left[ \begin{array}{rrcr} 1 & x_{1,1} & \cdots & x_{p,1} \\ \vdots & \vdots & & \vdots \\ 1 & x_{1,n} & \cdots & x_{p,n} \end{array} \right]. $$ Then you can show that $$ \widehat\beta = (X^T X)^{-1} X^T Y. $$ Often it is assumed that the "errors" in line $(1)$ above are independent random variables distributed as $\operatorname{Normal}(0,\sigma^2).$ Then one can deduce that \begin{align} \widehat\beta \sim {} & \operatorname{Normal}(\beta, \sigma^2(X^T X)^{-1}) \\ & \text{(a $p$-dimensional normal distribution} \tag 2 \\ & \phantom{(} \text{with a $(p+1)\times1$ expected value} \\ & \phantom{(} \text{and a $(p+1)\times(p+1)$ variance matrix.)} \\[10pt] & \frac 1 {\sigma^2}\sum_{k=1}^n (y_i - (\widehat\beta_0 + \widehat\beta_1 x_{1,i} + \cdots + \widehat\beta_p x_{p,i}))^2 \sim \chi^2_{n-p-1} \tag 3 \\[10pt] \text{and } & \text{lines $(2)$ and $3$ above are independent.} \tag 4 \end{align} By using $(2),$ $(3),$ and $(4)$ and the theory of the t-distribution, one can deduce confidence intervals for any $\beta_k$ for $k=0,\ldots,p.$

$\endgroup$
3
  • $\begingroup$ My answer initially had the wrong number of degrees of freedom for the chi-square random variable. It should have been $\chi^2_{n-p-1}.$ What I wrote would be correct only if $p-1.$ I've corrected it. $\endgroup$ Dec 18, 2017 at 19:05
  • $\begingroup$ Thanks for the answer. I would like to know more about 2 details. 1/ How do you deduce that the vector $\hat{\beta}$ follows a normal distribution just using the normal distribution of the error ? And how do you find its parameters. 2/ Could you give me some reference and/or link where I can find more details about using (2) and (3) 59 find the intervals for any $\beta_k$ ? Thanks ! $\endgroup$
    – mlx
    Dec 20, 2017 at 3:49
  • $\begingroup$ @MhamedJabri : Notice that $\widehat\beta$ is of the form $AY,$ where $A$ is an $n\times(p+1)$ matrix. If $Y\sim N_n(X\beta, \sigma^2 I_n)$ (an $n$-dimensional normal distribution) then $$ AY \sim N_{p+1} \left( AX\beta,\, A \Big( \sigma^2 I_n\Big) A^T \right). $$ The variance, $A \Big( \sigma^2 I_n\Big) A^T,$ is a $(p+1)\times(p+1)$ matrix. $\qquad$ $\endgroup$ Dec 20, 2017 at 19:04

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .