# Prove that the the variance estimator $\widehat{\sigma}^2=MSE/(n-2)$ is biased is the simple linear regression model

This is in scope of the simple linear model. Im trying to prove that $\mathbb{E}\left(\widehat{\sigma}^2\right) = \sigma^2$ for

$$\widehat{\sigma}^2 = \frac{1}{n-2}\sum^n_{i=1} \left(y_i-\widehat{y}_i\right)^2$$

where $$Y_i\sim N(\beta_0+\beta_1x_i,\sigma^2)$$ and $\widehat{y_i},i=1,2,3,...,n$ are pedricted values and $y_1,y_2,...,y_n$ is a sample from $Y_i$

$\newcommand{\b}{\begin{bmatrix}}\newcommand{\eb}{\end{bmatrix}}$ The vector of fitted values $$\hat Y = \b \hat y_1 \\ \vdots \\ \hat y_n\eb$$ is the orthogonal projection of $$Y = \b y_1 \\ \vdots \\ y_n \eb$$ onto the column space of the design matrix $$X = \b 1 & x_1 \\ \vdots & \vdots \\ 1 & x_n \eb.$$ The vector $\hat\varepsilon$ is $Y-\hat Y$. That means $\hat\varepsilon$ is the projection of $Y$ onto the $(n-2)$-dimensional orthogonal complement of that column space. Notice that this latter orthogonal projection maps the expected value $$\mathbb E Y = \b 1 & x_1 \\ \vdots & \vdots \\ 1 & x_n \eb \b \beta_0 \\ \beta_1 \eb$$ to the zero vector. Let $H$ (conventionally called the "hat matrix") be the matrix of the first orthogonal projection above, so that $\hat Y= HY$. Then $\varepsilon = (I-H)Y$. Recall that the $n\times n$ matrix $H$ of rank $2$ is $X(X^TX)^{-1}X^T$ and this is symmetric. It is also idempotent, i.e. $H^2=H$, or in other words, if you project onto a space, and then project that projection onto that same space, then you just get the point you had when you first projected onto the space. Similarly $I-H$ is symmetric and idempotent. So $$\varepsilon \sim N_n(0, (I-H)\Big(\sigma^2 I\Big)(I-H)^T) = N_n(0, \sigma^2(I-H)).$$
You have a normally distributed random vector in an $(n-2)$-dimensional Euclidean space. Its expected value is $0$ and the normal distribution is spherically symmetric. Hence the square of its norm is distributed as $\sigma^2\chi^2_{n-2}$, so the expected value of the square of the norm is $\sigma^2(n-2)$.
Indeed, if you project $Y$ onto that $(n-2)$-dimensional space, getting $\hat\varepsilon$, and then let $U_1,\ldots, U_{n-2}$ be the coordinates of $\hat\varepsilon$ with respect to an orthonormal basis of that space then you have $$U_1,\ldots,U_{n-2}\sim\mathrm{i.i.d.}\ N(0,\sigma^2)$$ and $$\varepsilon_1^2+\cdots+\varepsilon_n^2=U_1^2+\cdots+U_{n-2}^2.$$
• What would $H$ be in the simple case ? i.e n = 1 ?? – Teodor Fredriksson Sep 12 '14 at 21:18
• In case $n=1$, $H$ is just a $1\times1$ matrix whose only entry is $1$. You don't do regression with only one data point. And with only two data points with different $x$ values, the straight-line fit is perfect. Let's try $n=3$. You have three $x$ values: $1$, $2$, and $3$. Then $X=\begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}$, and $H=X(X^TX)^{-1}X^T= (1/6) \begin{bmatrix} 5 & 2 & -1 \\ 2 & 2 & 2 \\ -1 & 2 & 5 \end{bmatrix}$. This is a $3\times3$ symmetric idempotent matrix of rank $2$. ${}\qquad{}$ – Michael Hardy Sep 12 '14 at 22:44
• Call the $y$ values $y_1,y_2,y_3$. Then $HY=\begin{bmatrix} (5y_1+2y_2-y_3)/6 \\ (y_1+y_2+y_3)/3 \\ (-y_1+2y_2+5y_3)/6 \end{bmatrix}$. That is the vector $\widehat Y$ of fitted values. And $Y-\widehat Y$ is the vector $\widehat\varepsilon$ of residuals. ${}\qquad{}$ – Michael Hardy Sep 12 '14 at 22:46
• . . . . and may I add that if $n=1$ then the $n-2$ by which you divide is negative, and if $n=2$, then you'd be dividing by $0$. But if $n=2$ then the numerator would also be $0$, i.e. the sum of squares of residuals would be $0$ because the fit would be perfect. A jocular rule of experimentation is: if a straight-line fit is required, then obtain only two data points. ${}\qquad{}$ – Michael Hardy Sep 12 '14 at 22:57