I want find the covariance of the estimates $\hat{\beta_0}$ and $\hat{\beta_1}$. There are many answers such as this one that give the answer as \begin{align*} \operatorname{Cov}(\hat{\beta_0}, \hat{\beta_1}) = -\frac{\bar{x}\sigma^2}{n\sum(x_i - \bar{x})^2} \end{align*} where $\sigma^2$ is the variance of the errors.

However, when I try to get to this answer via non-matrix form for case of simple linear regression, I feel as if I'm breaking a few rules on notation and arithmetic. Since the only random variables are the $y_i$, we can bring outside of the covariance term all of the $x_i$ terms because they are treated as constants. For ease of notation, let $S_{xx} = \sum (x_i - \bar{x})^2$. \begin{align} \operatorname{Cov}(\hat{\beta_0}, \hat{\beta_1}) &= \operatorname{Cov} \Big(\bar{y}-\hat{\beta_1}\bar{x}, \frac{\sum _j(x_j - \bar{x})y_j}{S_{xx}}\Big)\\ &= \frac{1}{S_{xx}} \operatorname{Cov} \Big(\frac{1}{n} \sum_i y_i - \frac{\bar{x}\sum_i (x_i - \bar{x})y_i}{S_{xx}} , \sum_j (x_j - \bar{x})y_j\Big)\\ &= \frac{1}{ S_{xx}} \sum_j \Bigg[(x_j - \bar{x}) \cdot \operatorname{Cov}\Big( \frac{1}{n} \sum_i y_i - \frac{\bar{x}\sum_i (x_i - \bar{x})y_i}{S_{xx}} , y_j \Big)\Bigg]\\ &= \frac{1}{ S_{xx}} \sum_i \sum_j \Bigg[(x_j - \bar{x}) \cdot \operatorname{Cov}\Big( \frac{y_i}{n} - \frac{\bar{x}(x_i - \bar{x})y_i}{S_{xx}} , y_j \Big)\Bigg]\\ &= \frac{1}{ S_{xx}} \sum_i \sum_j \Bigg[(x_j - \bar{x}) \cdot \operatorname{Cov} \Big(\Big[ \frac{1}{n} - \frac{\bar{x}(x_i - \bar{x})}{S_{xx}}\Big] y_i , y_j \Big)\Bigg]\\ &= \frac{1}{ S_{xx}} \sum_i \sum_j \Bigg[\Big(\frac{1}{n} - \frac{\bar{x}(x_i - \bar{x})}{S_{xx}}\Big)\cdot (x_j - \bar{x}) \cdot \operatorname{Cov}( y_i , y_j )\Bigg]\\ &= \frac{1}{ S_{xx}} \sum_{i} \Bigg[\Big(\frac{(x_i - \bar{x})}{n} - \frac{\bar{x}(x_i - \bar{x})^2}{S_{xx}}\Big) \cdot Var(y_i)\Bigg]\\ &= \Bigg(\frac{\sum (x_i - \bar{x})}{n S_{xx}} - \frac{\bar{x}\sum (x_i - \bar{x})^2}{nS_{xx}^2}\Bigg) \sigma^2\\ &=\Bigg(0 - \frac{\bar{x}}{nS_{xx}}\Bigg) \sigma^2 \\ &= -\frac{\bar{x}\sigma^2}{nS_{xx}} \end{align} What I am confused about is when the term $\sum_j (x_j - \bar{x})$ is pulled out since it can be treated as a constant, but doesn't $\sum_j (x_j - \bar{x}) = 0$? If so, shouldn't the answer be $0$? If the way that I dealt with the $i$ and $j$ terms is problematic, I would be grateful to hear advice on notation. Also, should the $\sigma^2$ be $\hat{\sigma}^2$ since we don't actually know the true variance of $y_i$, just an estimated one that can be obtained from the mean-squared error, $\hat{\sigma}^2 = \sum (y_i - \hat{y}_i)^2/(n-2)$?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.