In the book "Introduction to Statistical Learning" page 66, there are formulas of the standard errors of the coefficient estimates $\hat{\beta}_0$ and $\hat{\beta}_1$. I know the proof of $SE(\hat{\beta}_1)$ but I am confused about how to derive the formula for $$SE(\hat{\beta}_0)^2 = \sigma^2\left[\frac{1}{n} + \frac{\bar{x}^2}{{\sum_{i=1}^n}(x_i - \bar{x})^2}\right]$$ since $\sigma^2 = Var(\epsilon)$, not the variance of $y_i's$.

My calculation so far is as follows: $$Var(\hat{\beta}_0) = Var(\bar{y} - \hat{\beta}_1\bar{x}) = Var(\bar{y}) + \bar{x}^2\frac{\sigma^2}{{\sum_{i=1}^n}(x_i - \bar{x})^2} - 2\bar{x} Cov(\bar{y}, \hat{\beta}_1) $$in which $\sigma^2 = Var(\epsilon)$.

$Cov(\bar{y}, \hat{\beta}_1) = 0$ since $\bar{y}$ and $\hat{\beta}_1$ are uncorrelated. $Var(\bar{y}) = \frac{\sigma^2}{n}$ in which $\sigma^2 = Var(y_i)$.

So how can we have the formula for $SE(\hat{\beta}_0)^2$ as above since the 2 $\sigma's$ are different from each other?

Many thanks in advance for your help!

  • $\begingroup$ If you know that $Var( \hat \beta_1)=\frac{\sigma^2}{{\sum_{i=1}^n}(x_i - \bar{x})^2}$ and you use $Var(\hat \beta_1\cdot \overline x)=\overline x^2\cdot Var(\hat \beta_1)$ the formula for $Var( \hat \beta_0)$ follows straightforward. I don´t see any difference. $\endgroup$ – callculus Oct 15 '18 at 19:16
  • $\begingroup$ But they are the same $Var(y_i) =Var(\beta_0 + \beta_1 x_i +\epsilon_i) =0 + Var(\epsilon_i)$ $\endgroup$ – papasmurfete Oct 15 '18 at 19:29
  • $\begingroup$ @callculus But $SE(\hat{\beta_0})^2 = \sigma^2[\frac{1}{n} + \frac{\bar{x}^2}{{\sum_{i=1}^n}(x_i - \bar{x})^2}]$, not just $\overline x^2\cdot Var(\hat \beta_1)$. There is also a component of $\sigma^2\frac{1}{n}$. $\endgroup$ – Sophil Oct 15 '18 at 21:08
  • 1
    $\begingroup$ In linear regression, explanatory variables $X$ (or independent variables) are not random variables, neither (the real value of) $\beta_0$ and $\beta_1$ . $\endgroup$ – papasmurfete Oct 16 '18 at 6:35
  • 1
    $\begingroup$ In the linear regression $y_1,y_2,\cdots,y_n $ are independent with $y_i\sim N(\beta_0+\beta_1 x_i,\sigma^2)$ so $\overline{y}\sim N(\beta_0+\beta_1\bar x,\sigma^2/n)$, without the use of CLT. But in many cases when n is big we can assume Normal distribution due to CLT. $\endgroup$ – papasmurfete Oct 16 '18 at 7:26

You have found out that

$$Var(\hat{\beta}_0) = Var(\bar{y} - \hat{\beta}_1\bar{x}) = Var(\bar{y}) + \bar{x}^2\frac{\sigma^2}{{\sum_{i=1}^n}(x_i - \bar{x})^2} - 2\bar{x} \underbrace{Cov(\bar{y}, \hat{\beta}_1)}_{=0}$$ in which $Var(\bar{y}) =\frac{\sigma^2}{n}$.

Inserting $\frac{\sigma^2}{n}$ for $Var(\bar{y}) $

$$Var(\hat{\beta}_0) = \frac{\sigma^2}{n} + \bar{x}^2\frac{\sigma^2}{{\sum_{i=1}^n}(x_i - \bar{x})^2}=\sigma^2 \cdot \left(\frac{1}{n} + \bar{x}^2\frac{1}{{\sum_{i=1}^n}(x_i - \bar{x})^2} \right)$$

  • $\begingroup$ Thanks for your answer! Actually, my question is how we can put the common factor $\sigma^2$ while one is the variance of the error terms $\epsilon$ (in the formula for $\hat{\beta}_1$) and the other in $Var(\bar{y)}$ is the variance of $y_i's$. Maybe I wasn't clear in my question. My concern has been answered by papasmurfete, which is $Var(\bar{y}) = Var(\epsilon)$. $\endgroup$ – Sophil Oct 16 '18 at 6:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.