This is in scope of the simple linear model. Im trying to prove that $\mathbb{E}\left(\widehat{\sigma}^2\right) = \sigma^2$ for

$$\widehat{\sigma}^2 = \frac{1}{n-2}\sum^n_{i=1} \left(y_i-\widehat{y}_i\right)^2$$

where $$Y_i\sim N(\beta_0+\beta_1x_i,\sigma^2)$$ and $\widehat{y_i},i=1,2,3,...,n$ are pedricted values and $y_1,y_2,...,y_n$ is a sample from $Y_i$


$\newcommand{\b}{\begin{bmatrix}}\newcommand{\eb}{\end{bmatrix}}$ The vector of fitted values $$ \hat Y = \b \hat y_1 \\ \vdots \\ \hat y_n\eb $$ is the orthogonal projection of $$ Y = \b y_1 \\ \vdots \\ y_n \eb $$ onto the column space of the design matrix $$ X = \b 1 & x_1 \\ \vdots & \vdots \\ 1 & x_n \eb. $$ The vector $\hat\varepsilon$ is $Y-\hat Y$. That means $\hat\varepsilon$ is the projection of $Y$ onto the $(n-2)$-dimensional orthogonal complement of that column space. Notice that this latter orthogonal projection maps the expected value $$ \mathbb E Y = \b 1 & x_1 \\ \vdots & \vdots \\ 1 & x_n \eb \b \beta_0 \\ \beta_1 \eb $$ to the zero vector. Let $H$ (conventionally called the "hat matrix") be the matrix of the first orthogonal projection above, so that $\hat Y= HY$. Then $\varepsilon = (I-H)Y$. Recall that the $n\times n$ matrix $H$ of rank $2$ is $X(X^TX)^{-1}X^T$ and this is symmetric. It is also idempotent, i.e. $H^2=H$, or in other words, if you project onto a space, and then project that projection onto that same space, then you just get the point you had when you first projected onto the space. Similarly $I-H$ is symmetric and idempotent. So $$ \varepsilon \sim N_n(0, (I-H)\Big(\sigma^2 I\Big)(I-H)^T) = N_n(0, \sigma^2(I-H)). $$

You have a normally distributed random vector in an $(n-2)$-dimensional Euclidean space. Its expected value is $0$ and the normal distribution is spherically symmetric. Hence the square of its norm is distributed as $\sigma^2\chi^2_{n-2}$, so the expected value of the square of the norm is $\sigma^2(n-2)$.

Indeed, if you project $Y$ onto that $(n-2)$-dimensional space, getting $\hat\varepsilon$, and then let $U_1,\ldots, U_{n-2}$ be the coordinates of $\hat\varepsilon$ with respect to an orthonormal basis of that space then you have $$ U_1,\ldots,U_{n-2}\sim\mathrm{i.i.d.}\ N(0,\sigma^2) $$ and $$\varepsilon_1^2+\cdots+\varepsilon_n^2=U_1^2+\cdots+U_{n-2}^2.$$

  • $\begingroup$ What would $H$ be in the simple case ? i.e n = 1 ?? $\endgroup$ – Teodor Fredriksson Sep 12 '14 at 21:18
  • $\begingroup$ In case $n=1$, $H$ is just a $1\times1$ matrix whose only entry is $1$. You don't do regression with only one data point. And with only two data points with different $x$ values, the straight-line fit is perfect. Let's try $n=3$. You have three $x$ values: $1$, $2$, and $3$. Then $X=\begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}$, and $H=X(X^TX)^{-1}X^T= (1/6) \begin{bmatrix} 5 & 2 & -1 \\ 2 & 2 & 2 \\ -1 & 2 & 5 \end{bmatrix}$. This is a $3\times3$ symmetric idempotent matrix of rank $2$. ${}\qquad{}$ $\endgroup$ – Michael Hardy Sep 12 '14 at 22:44
  • $\begingroup$ Call the $y$ values $y_1,y_2,y_3$. Then $HY=\begin{bmatrix} (5y_1+2y_2-y_3)/6 \\ (y_1+y_2+y_3)/3 \\ (-y_1+2y_2+5y_3)/6 \end{bmatrix}$. That is the vector $\widehat Y$ of fitted values. And $Y-\widehat Y$ is the vector $\widehat\varepsilon$ of residuals. ${}\qquad{}$ $\endgroup$ – Michael Hardy Sep 12 '14 at 22:46
  • $\begingroup$ . . . . and may I add that if $n=1$ then the $n-2$ by which you divide is negative, and if $n=2$, then you'd be dividing by $0$. But if $n=2$ then the numerator would also be $0$, i.e. the sum of squares of residuals would be $0$ because the fit would be perfect. A jocular rule of experimentation is: if a straight-line fit is required, then obtain only two data points. ${}\qquad{}$ $\endgroup$ – Michael Hardy Sep 12 '14 at 22:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.