3
$\begingroup$

I have a linear regression model $\hat{y_i}=\hat{\beta_0}+\hat{\beta_1}x_i+\hat{\epsilon_i}$, where $\hat{\beta_0}$ and $\hat{\beta_1}$ are normally distributed unbiased estimators, and $\hat{\epsilon_i}$ is Normal with mean $0$ and variance $\sigma^2$. I need to show that

$$\operatorname{Var}\left(\hat{\beta_0}\right)=\frac{\sigma^2\sum_{i=1}^nx_i^2}{n\sum_{i=1}^n\left(x_i-\bar{x}\right)^2}$$

$$\operatorname{Var}\left(\hat{\beta_1}\right)=\frac{\sigma^2}{\sum_{i=1}^n\left(x_i-\bar{x}\right)^2}$$

and

$$\operatorname{cov}\left(\hat{\beta_0},\hat{\beta_1}\right)=\frac{-\sigma^2\sum_{i=1}^nx_i}{n\sum_{i=1}^n\left(x_i-\bar{x}\right)^2}$$

Can anyone help me out?

Thanks.

Improve this question
$\endgroup$
  • $\begingroup$ I used the notation $u$ instead of $\epsilon$ for the errors. Also you should write your model without the $\hat{\epsilon_i}$ terms. If you are ok with the answer, you can upvote and accept it, so that your question can be considered answered. $\endgroup$ – Jimmy R. Feb 23 '14 at 18:10
  • $\begingroup$ There is an error in the question: The variance of $\varepsilon_i$ is normally assumed to be $N(0,\sigma^2$, but that of $\hat\varepsilon_i$ is more complicated. $\endgroup$ – Michael Hardy Feb 23 '14 at 18:51
  • $\begingroup$ The model should say $y_i = \beta_0+\beta_1 x_i + \varepsilon_i$, not $\hat y_i = \hat\beta_0+\hat\beta_1 x_i + \hat\varepsilon_i$. In standard usage, it would be true that $\hat y_i = \hat\beta_0+\hat\beta_1 x_i$, WITHOUT any $\varepsilon$ term, and $\hat\varepsilon_i=y_i-(\hat\beta_0+\hat\beta_1 x_i)$ $=y_i-\hat y_i$. But the model itself should be stated without any "hats". It you're thinking about this kind of problem, you should be much more careful with things like this. $\endgroup$ – Michael Hardy Feb 23 '14 at 18:55
4
$\begingroup$

From the least squares estimation method, we know that $$\hat{\beta}=(X'X)^{-1}X'Y$$ and that $\hat{\beta}$ is an unbiased estimator of $\beta$, i.e $E[\hat{\beta}]=\beta$. Moreover, the linear model $$\begin{equation} Y=X\beta +u \end{equation}$$ has the assumption that $$Y\sim N(\mu=\beta_0+\beta_1x,\sigma)$$ or equivalently that $u \sim N(\mu=0,\sigma)$. Based on the above we can prove all three results (simultaneously) by calculating the variance-covariance matrix of $b$ which is equal to: $$Var(\hat{\beta)}:=\sigma^2(\hat{\beta})=\left( \begin{array}{ccc} Var(\hat{\beta_0}) & Cov(\hat{\beta_0},\hat{\beta_1}) \\ Cov(\hat{\beta_0},\hat{\beta_1}) & Var(\hat{\beta_1}) \end{array} \right)$$ By the properties of variance we have that

$$\begin{align*}Var(\hat{\beta})&=E[\hat{\beta}\phantom{}^2]-E[\hat{\beta}]^2=E[((X'X)^{-1}X'Y)^2]-\beta^2=E[((X'X)^{-1}X'(X\beta +u))^2]-\beta^2=\\&=E[((X'X)^{-1}X'X\beta +(X'X)^{-1}X'u))^2]-\beta^2=E[(\beta+(X'X)^{-1}X'u))^2]-\beta^2=\\&=E[\beta^2]+2(X'X)^{-1}X'E[u]+E[((X'X)^{-1}X'u))^2]-\beta^2=\\&=\beta^2+0+E[((X'X)^{-1}X'u))^2]-\beta^2=E[((X'X)^{-1}X'u))^2]=\\&=\left((X'X)^{-1}X'\right)^2\cdot E[u^2]\end{align*}$$But, since $E[u]=0$ we have that $E[u^2]=Var(u)=\sigma^2$ and by substituting in the above equation we find that $$\begin{align*}Var(\hat{\beta})&=\left((X'X)^{-1}X'\right)^2\cdot E[u^2]=(X'X)^{-1}X'\cdot(X'X)^{-1}X'\cdot\sigma^2=\sigma^2(X'X)^{-1}\cdot I=\\&=\sigma^2(X'X)^{-1}.\end{align*}$$ Now, since $$(X'X)^{-1}=\left( \begin{array}{ccc} \frac{\sum x_i^2}{n\sum (x_1-\bar{x})^2} & \frac{-\sum x_i}{n\sum (x_1-\bar{x})^2} \\ \frac{-\sum x_i}{n\sum (x_1-\bar{x})^2} & \frac{1}{\sum (x_1-\bar{x})^2} \end{array} \right)$$ (which is also known or can be easily derived algebraically) you have the result that: $$\begin{align*} Var(\hat{\beta})&=\left( \begin{array}{ccc} Var(\hat{\beta_0}) & Cov(\hat{\beta_0},\hat{\beta_1}) \\ Cov(\hat{\beta_0},\hat{\beta_1} & Var(\hat{\beta_1}) \end{array} \right)=\sigma^2\left(X'X\right)^{-1}=\\&\phantom{kl}\\&=\left( \begin{array}{ccc} \frac{\sigma^2 \sum x_i^2}{n\sum (x_1-\bar{x})^2} & \frac{-\sigma^2 \sum x_i}{n\sum (x_1-\bar{x})^2} \\ \frac{-\sigma^2 \sum x_i}{n\sum (x_1-\bar{x})^2} & \frac{\sigma^2}{\sum (x_1-\bar{x})^2} \end{array} \right) \end{align*}$$

Improve this answer
$\endgroup$
  • $\begingroup$ The model is wrong here. You've got $\beta X$ where you should have $X\beta$. You need to be more careful with matrices. $\endgroup$ – Michael Hardy Feb 23 '14 at 18:52
  • 1
    $\begingroup$ To find $\operatorname{var}\hat\beta$, I would just write $\operatorname{var}\hat\beta=\operatorname{var}((X'X)^{-1}X'Y)$ $=\Big((X'X)^{-1}X'\Big)\operatorname{var}(Y)\Big((X'X)^{-1}X'\Big)'$ $=\Big((X'X)^{-1}X'\Big)\operatorname{var}(Y)\Big((X(X'X)^{-1}\Big)$ and go on from there, recalling that $\operatorname{var}(Y)$ is just $\sigma^2$ times the $n\times n$ identity matrix. $\endgroup$ – Michael Hardy Feb 23 '14 at 19:00
  • $\begingroup$ @MichaelHardy. Thanks for the comment, I corrected the model. Your approach is also equally correct and admittedly faster. $\endgroup$ – Jimmy R. Feb 23 '14 at 19:05
  • $\begingroup$ I think there is a mistake in the $X^\top X$ matrices: entry $[2,2]$ should be $\frac{1}{\sum(x_i - \bar x)^2}$, and the matrix would be $\begin{bmatrix}\frac{\sum x_i^2}{n\sum(x_i - \bar x)^2} & \frac{-\sum x_i}{n\sum(x_i - \bar x)^2}\\\frac{-\sum x_i}{n\sum(x_i - \bar x)^2}& \frac{1}{\sum(x_i - \bar x)^2}\end{bmatrix}$ $\endgroup$ – Antoni Parellada Jan 28 '17 at 2:25
  • $\begingroup$ @AntoniParellada They should introduce a badge for things like this "found a mistake, 3 years later in a upvoted question" or something like that :) $\endgroup$ – Jimmy R. Jan 29 '17 at 7:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.