# Variance of Coefficients in a Simple Linear Regression

I have a linear regression model $\hat{y_i}=\hat{\beta_0}+\hat{\beta_1}x_i+\hat{\epsilon_i}$, where $\hat{\beta_0}$ and $\hat{\beta_1}$ are normally distributed unbiased estimators, and $\hat{\epsilon_i}$ is Normal with mean $0$ and variance $\sigma^2$. I need to show that

$$\operatorname{Var}\left(\hat{\beta_0}\right)=\frac{\sigma^2\sum_{i=1}^nx_i^2}{n\sum_{i=1}^n\left(x_i-\bar{x}\right)^2}$$

$$\operatorname{Var}\left(\hat{\beta_1}\right)=\frac{\sigma^2}{\sum_{i=1}^n\left(x_i-\bar{x}\right)^2}$$

and

$$\operatorname{cov}\left(\hat{\beta_0},\hat{\beta_1}\right)=\frac{-\sigma^2\sum_{i=1}^nx_i}{n\sum_{i=1}^n\left(x_i-\bar{x}\right)^2}$$

Can anyone help me out?

Thanks.

• I used the notation $u$ instead of $\epsilon$ for the errors. Also you should write your model without the $\hat{\epsilon_i}$ terms. If you are ok with the answer, you can upvote and accept it, so that your question can be considered answered. Feb 23 '14 at 18:10
• There is an error in the question: The variance of $\varepsilon_i$ is normally assumed to be $N(0,\sigma^2$, but that of $\hat\varepsilon_i$ is more complicated. Feb 23 '14 at 18:51
• The model should say $y_i = \beta_0+\beta_1 x_i + \varepsilon_i$, not $\hat y_i = \hat\beta_0+\hat\beta_1 x_i + \hat\varepsilon_i$. In standard usage, it would be true that $\hat y_i = \hat\beta_0+\hat\beta_1 x_i$, WITHOUT any $\varepsilon$ term, and $\hat\varepsilon_i=y_i-(\hat\beta_0+\hat\beta_1 x_i)$ $=y_i-\hat y_i$. But the model itself should be stated without any "hats". It you're thinking about this kind of problem, you should be much more careful with things like this. Feb 23 '14 at 18:55

From the least squares estimation method, we know that $$\hat{\beta}=(X'X)^{-1}X'Y$$ and that $\hat{\beta}$ is an unbiased estimator of $\beta$, i.e $E[\hat{\beta}]=\beta$. Moreover, the linear model $$$$Y=X\beta +u$$$$ has the assumption that $$Y\sim N(\mu=\beta_0+\beta_1x,\sigma)$$ or equivalently that $u \sim N(\mu=0,\sigma)$. Based on the above we can prove all three results (simultaneously) by calculating the variance-covariance matrix of $b$ which is equal to: $$Var(\hat{\beta)}:=\sigma^2(\hat{\beta})=\left( \begin{array}{ccc} Var(\hat{\beta_0}) & Cov(\hat{\beta_0},\hat{\beta_1}) \\ Cov(\hat{\beta_0},\hat{\beta_1}) & Var(\hat{\beta_1}) \end{array} \right)$$ By the properties of variance we have that
\begin{align*}Var(\hat{\beta})&=E[\hat{\beta}\phantom{}^2]-E[\hat{\beta}]^2=E[((X'X)^{-1}X'Y)^2]-\beta^2=E[((X'X)^{-1}X'(X\beta +u))^2]-\beta^2=\\&=E[((X'X)^{-1}X'X\beta +(X'X)^{-1}X'u))^2]-\beta^2=E[(\beta+(X'X)^{-1}X'u))^2]-\beta^2=\\&=E[\beta^2]+2(X'X)^{-1}X'E[u]+E[((X'X)^{-1}X'u))^2]-\beta^2=\\&=\beta^2+0+E[((X'X)^{-1}X'u))^2]-\beta^2=E[((X'X)^{-1}X'u))^2]=\\&=\left((X'X)^{-1}X'\right)^2\cdot E[u^2]\end{align*}But, since $E[u]=0$ we have that $E[u^2]=Var(u)=\sigma^2$ and by substituting in the above equation we find that \begin{align*}Var(\hat{\beta})&=\left((X'X)^{-1}X'\right)^2\cdot E[u^2]=(X'X)^{-1}X'\cdot(X'X)^{-1}X'\cdot\sigma^2=\sigma^2(X'X)^{-1}\cdot I=\\&=\sigma^2(X'X)^{-1}.\end{align*} Now, since $$(X'X)^{-1}=\left( \begin{array}{ccc} \frac{\sum x_i^2}{n\sum (x_1-\bar{x})^2} & \frac{-\sum x_i}{n\sum (x_1-\bar{x})^2} \\ \frac{-\sum x_i}{n\sum (x_1-\bar{x})^2} & \frac{1}{\sum (x_1-\bar{x})^2} \end{array} \right)$$ (which is also known or can be easily derived algebraically) you have the result that: \begin{align*} Var(\hat{\beta})&=\left( \begin{array}{ccc} Var(\hat{\beta_0}) & Cov(\hat{\beta_0},\hat{\beta_1}) \\ Cov(\hat{\beta_0},\hat{\beta_1} & Var(\hat{\beta_1}) \end{array} \right)=\sigma^2\left(X'X\right)^{-1}=\\&\phantom{kl}\\&=\left( \begin{array}{ccc} \frac{\sigma^2 \sum x_i^2}{n\sum (x_1-\bar{x})^2} & \frac{-\sigma^2 \sum x_i}{n\sum (x_1-\bar{x})^2} \\ \frac{-\sigma^2 \sum x_i}{n\sum (x_1-\bar{x})^2} & \frac{\sigma^2}{\sum (x_1-\bar{x})^2} \end{array} \right) \end{align*}
• The model is wrong here. You've got $\beta X$ where you should have $X\beta$. You need to be more careful with matrices. Feb 23 '14 at 18:52
• To find $\operatorname{var}\hat\beta$, I would just write $\operatorname{var}\hat\beta=\operatorname{var}((X'X)^{-1}X'Y)$ $=\Big((X'X)^{-1}X'\Big)\operatorname{var}(Y)\Big((X'X)^{-1}X'\Big)'$ $=\Big((X'X)^{-1}X'\Big)\operatorname{var}(Y)\Big((X(X'X)^{-1}\Big)$ and go on from there, recalling that $\operatorname{var}(Y)$ is just $\sigma^2$ times the $n\times n$ identity matrix. Feb 23 '14 at 19:00
• I think there is a mistake in the $X^\top X$ matrices: entry $[2,2]$ should be $\frac{1}{\sum(x_i - \bar x)^2}$, and the matrix would be $\begin{bmatrix}\frac{\sum x_i^2}{n\sum(x_i - \bar x)^2} & \frac{-\sum x_i}{n\sum(x_i - \bar x)^2}\\\frac{-\sum x_i}{n\sum(x_i - \bar x)^2}& \frac{1}{\sum(x_i - \bar x)^2}\end{bmatrix}$ Jan 28 '17 at 2:25