7
$\begingroup$

I have a linear regression model $\hat{y_i}=\hat{\beta_0}+\hat{\beta_1}x_i+\hat{\epsilon_i}$, where $\hat{\beta_0}$ and $\hat{\beta_1}$ are normally distributed unbiased estimators, and $\hat{\epsilon_i}$ is Normal with mean $0$ and variance $\sigma^2$. I need to show that

$$\operatorname{Var}\left(\hat{\beta_0}\right)=\frac{\sigma^2\sum_{i=1}^nx_i^2}{n\sum_{i=1}^n\left(x_i-\bar{x}\right)^2}$$

$$\operatorname{Var}\left(\hat{\beta_1}\right)=\frac{\sigma^2}{\sum_{i=1}^n\left(x_i-\bar{x}\right)^2}$$

and

$$\operatorname{cov}\left(\hat{\beta_0},\hat{\beta_1}\right)=\frac{-\sigma^2\sum_{i=1}^nx_i}{n\sum_{i=1}^n\left(x_i-\bar{x}\right)^2}$$

Can anyone help me out?

Thanks.

$\endgroup$
3
  • $\begingroup$ I used the notation $u$ instead of $\epsilon$ for the errors. Also you should write your model without the $\hat{\epsilon_i}$ terms. If you are ok with the answer, you can upvote and accept it, so that your question can be considered answered. $\endgroup$
    – Jimmy R.
    Feb 23, 2014 at 18:10
  • $\begingroup$ There is an error in the question: The variance of $\varepsilon_i$ is normally assumed to be $N(0,\sigma^2$, but that of $\hat\varepsilon_i$ is more complicated. $\endgroup$ Feb 23, 2014 at 18:51
  • $\begingroup$ The model should say $y_i = \beta_0+\beta_1 x_i + \varepsilon_i$, not $\hat y_i = \hat\beta_0+\hat\beta_1 x_i + \hat\varepsilon_i$. In standard usage, it would be true that $\hat y_i = \hat\beta_0+\hat\beta_1 x_i$, WITHOUT any $\varepsilon$ term, and $\hat\varepsilon_i=y_i-(\hat\beta_0+\hat\beta_1 x_i)$ $=y_i-\hat y_i$. But the model itself should be stated without any "hats". It you're thinking about this kind of problem, you should be much more careful with things like this. $\endgroup$ Feb 23, 2014 at 18:55

1 Answer 1

4
$\begingroup$

From the least squares estimation method, we know that $$\hat{\beta}=(X'X)^{-1}X'Y$$ and that $\hat{\beta}$ is an unbiased estimator of $\beta$, i.e $E[\hat{\beta}]=\beta$. Moreover, the linear model $$\begin{equation} Y=X\beta +u \end{equation}$$ has the assumption that $$Y\sim N(\mu=\beta_0+\beta_1x,\sigma)$$ or equivalently that $u \sim N(\mu=0,\sigma)$. Based on the above we can prove all three results (simultaneously) by calculating the variance-covariance matrix of $b$ which is equal to: $$Var(\hat{\beta)}:=\sigma^2(\hat{\beta})=\left( \begin{array}{ccc} Var(\hat{\beta_0}) & Cov(\hat{\beta_0},\hat{\beta_1}) \\ Cov(\hat{\beta_0},\hat{\beta_1}) & Var(\hat{\beta_1}) \end{array} \right)$$ By the properties of variance we have that

$$\begin{align*}Var(\hat{\beta})&=E[\hat{\beta}\phantom{}^2]-E[\hat{\beta}]^2=E[((X'X)^{-1}X'Y)^2]-\beta^2=E[((X'X)^{-1}X'(X\beta +u))^2]-\beta^2=\\&=E[((X'X)^{-1}X'X\beta +(X'X)^{-1}X'u))^2]-\beta^2=E[(\beta+(X'X)^{-1}X'u))^2]-\beta^2=\\&=E[\beta^2]+2(X'X)^{-1}X'E[u]+E[((X'X)^{-1}X'u))^2]-\beta^2=\\&=\beta^2+0+E[((X'X)^{-1}X'u))^2]-\beta^2=E[((X'X)^{-1}X'u))^2]=\\&=\left((X'X)^{-1}X'\right)^2\cdot E[u^2]\end{align*}$$But, since $E[u]=0$ we have that $E[u^2]=Var(u)=\sigma^2$ and by substituting in the above equation we find that $$\begin{align*}Var(\hat{\beta})&=\left((X'X)^{-1}X'\right)^2\cdot E[u^2]=(X'X)^{-1}X'\cdot(X'X)^{-1}X'\cdot\sigma^2=\sigma^2(X'X)^{-1}\cdot I=\\&=\sigma^2(X'X)^{-1}.\end{align*}$$ Now, since $$(X'X)^{-1}=\left( \begin{array}{ccc} \frac{\sum x_i^2}{n\sum (x_1-\bar{x})^2} & \frac{-\sum x_i}{n\sum (x_1-\bar{x})^2} \\ \frac{-\sum x_i}{n\sum (x_1-\bar{x})^2} & \frac{1}{\sum (x_1-\bar{x})^2} \end{array} \right)$$ (which is also known or can be easily derived algebraically) you have the result that: $$\begin{align*} Var(\hat{\beta})&=\left( \begin{array}{ccc} Var(\hat{\beta_0}) & Cov(\hat{\beta_0},\hat{\beta_1}) \\ Cov(\hat{\beta_0},\hat{\beta_1} & Var(\hat{\beta_1}) \end{array} \right)=\sigma^2\left(X'X\right)^{-1}=\\&\phantom{kl}\\&=\left( \begin{array}{ccc} \frac{\sigma^2 \sum x_i^2}{n\sum (x_1-\bar{x})^2} & \frac{-\sigma^2 \sum x_i}{n\sum (x_1-\bar{x})^2} \\ \frac{-\sigma^2 \sum x_i}{n\sum (x_1-\bar{x})^2} & \frac{\sigma^2}{\sum (x_1-\bar{x})^2} \end{array} \right) \end{align*}$$

$\endgroup$
5
  • $\begingroup$ The model is wrong here. You've got $\beta X$ where you should have $X\beta$. You need to be more careful with matrices. $\endgroup$ Feb 23, 2014 at 18:52
  • 2
    $\begingroup$ To find $\operatorname{var}\hat\beta$, I would just write $\operatorname{var}\hat\beta=\operatorname{var}((X'X)^{-1}X'Y)$ $=\Big((X'X)^{-1}X'\Big)\operatorname{var}(Y)\Big((X'X)^{-1}X'\Big)'$ $=\Big((X'X)^{-1}X'\Big)\operatorname{var}(Y)\Big((X(X'X)^{-1}\Big)$ and go on from there, recalling that $\operatorname{var}(Y)$ is just $\sigma^2$ times the $n\times n$ identity matrix. $\endgroup$ Feb 23, 2014 at 19:00
  • $\begingroup$ @MichaelHardy. Thanks for the comment, I corrected the model. Your approach is also equally correct and admittedly faster. $\endgroup$
    – Jimmy R.
    Feb 23, 2014 at 19:05
  • $\begingroup$ I think there is a mistake in the $X^\top X$ matrices: entry $[2,2]$ should be $\frac{1}{\sum(x_i - \bar x)^2}$, and the matrix would be $\begin{bmatrix}\frac{\sum x_i^2}{n\sum(x_i - \bar x)^2} & \frac{-\sum x_i}{n\sum(x_i - \bar x)^2}\\\frac{-\sum x_i}{n\sum(x_i - \bar x)^2}& \frac{1}{\sum(x_i - \bar x)^2}\end{bmatrix}$ $\endgroup$ Jan 28, 2017 at 2:25
  • $\begingroup$ @AntoniParellada They should introduce a badge for things like this "found a mistake, 3 years later in a upvoted question" or something like that :) $\endgroup$
    – Jimmy R.
    Jan 29, 2017 at 7:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .