2
$\begingroup$

Suppose that $y = \beta_0+\beta_1x + \epsilon$ where $\operatorname{E}[\epsilon\mid X]=0$ and $\operatorname{Var}[\epsilon\mid X] = \sigma^2$.

In what special case is $\hat{\epsilon_i}^2$ an unbiased estimator for $\sigma^2$? Is $\hat{\epsilon_i}^2$ a consistent estimator for $\sigma^2$?

I have proved that $$E[\hat{\epsilon_i}^2] = \sigma^2\left(1-\frac{1}{n}- \frac{(x_i -\bar{x})^2}{\sum_{j=1}^n (x_j -\bar{x})^2} \right)$$ But I don't know how it can be unbiased or maybe I have proved wrong?

My proof: \begin{equation} \begin{split} \nonumber \hat{\epsilon_i} & = y_i - \hat{y_i} \\ & = \beta_0 + \beta_1 x_i + \epsilon_i - \hat{\beta_0} - \hat{\beta_1}x_i \\ & = \epsilon_i + (\beta_0 - \hat{\beta_0}) + (\beta_1 - \hat{\beta_1 })x_i \\ \hat{\beta_0} &= \bar{y} - \hat{\beta_1} \bar{x} \\ & = \frac{1}{n} \sum_{i=1}^{n}(\beta_0 + \beta_1 x_i + \epsilon_i) - \hat{\beta_1} \bar{x} \\ & = \beta_0 + (\beta_1 - \hat{\beta_1})\bar{x} + \frac{1}{n} \sum_{i=1}^{n} \epsilon_i \\ \hat{\beta_1} &= \frac{\sum (x_i - \bar{x}) y_i}{\sum (x_i - \bar{x})^2} \\ & = \frac{\sum (x_i - \bar{x})(\beta_0 + \beta_1 x_i + \epsilon_i)}{\sum (x_i - \bar{x})^2} \\ & = \beta_1 + \frac{\sum (x_i - \bar{x})\epsilon_i}{\sum (x_i - \bar{x})^2} \\ \therefore \hat{\epsilon_i} &= \epsilon_i - \frac{1}{n}\sum_{j=1}^{n} \epsilon_j + (\beta_1 - \hat{\beta_1})(x_i - \bar{x}) \\ & = \epsilon_i - \bar{\epsilon} - \frac{\sum_{j=1}^{n} (x_j - \bar{x})\epsilon_j}{\sum_{j=1}^{n} (x_j - \bar{x})^2} (x_i - \bar{x}) \\ \therefore \hat{\epsilon_i}^2 & = \epsilon_i^2 + \bar{\epsilon}^2 +\frac{(x_i -\bar{x})^2}{[\sum_{j=1}^{n} (x_j -\bar{x})^2]^2} [\sum_{j=1}^{n} (x_j -\bar{x}) \epsilon_j]^2 \\ & -2 \epsilon_i \bar{\epsilon} - 2\epsilon_i \frac{x_i - \bar{x}}{\sum_{j=1}^{n}(x_j - \bar{x})^2} \sum_{j=1}^{n} (x_j - \bar{x}) \epsilon_j + 2 \bar{\epsilon} \frac{x_i - \bar{x}}{\sum_{j=1}^{n}(x_j - \bar{x})^2} \sum_{j=1}^{n} (x_j - \bar{x}) \epsilon_j \\ \end{split} \end{equation} Now take expectation of the six parts of $\hat{\epsilon_i}^2$ given X (conditional symbol is omitted): \begin{equation} \begin{split} \nonumber (1) \ E[\epsilon_i^2] &= \sigma^2 \\ (2) \ E[\bar{\epsilon}^2] &= \frac{\sigma^2}{n} \\ (3) \ E[\cdot] & = \frac{(x_i -\bar{x})^2}{[\sum_{j=1}^{n} (x_j -\bar{x})^2]^2} E\left[[\sum_{j=1}^{n} (x_j -\bar{x}) \epsilon_j]^2 \right] \\ &= \frac{(x_i -\bar{x})^2}{[\sum_{j=1}^{n} (x_j -\bar{x})^2]^2} E\left[ \sum_{j=1}^{n}(x_j-\bar{x})^2 \epsilon_j^2 + 2 \sum_{j=1}^{n-1} \sum_{k=j+1}^{n} (x_j-\bar{x})\epsilon_j (x_k-\bar{x})\epsilon_k \right] \\ & = \frac{(x_i -\bar{x})^2 \sigma^2}{\sum_{j=1}^{n} (x_j -\bar{x})^2} \\ (4) \ E[-2\epsilon_i \bar{\epsilon}] & = -\frac{2}{n} E[\epsilon_i \sum_{j=1}^{n} \epsilon_j] \\ & = -\frac{2\sigma^2}{n} \\ (5) \ E[\cdot] &= -2 \frac{x_i - \bar{x}}{\sum_{j=1}^{n}(x_j - \bar{x})^2} E\left[ \epsilon_i \sum_{j=1}^{n} (x_j - \bar{x}) \epsilon_j\right] \\ & = -2 \frac{(x_i - \bar{x})^2 \sigma^2}{\sum_{j=1}^{n}(x_j - \bar{x})^2} \\ (6) \ E[\cdot] &= 2 \frac{x_i - \bar{x}}{\sum_{j=1}^{n}(x_j - \bar{x})^2} E\left[ \bar{\epsilon} \sum_{j=1}^{n} (x_j - \bar{x}) \epsilon_j\right] \\ & = 2 \frac{x_i - \bar{x}}{\sum_{j=1}^{n}(x_j - \bar{x})^2} \frac{1}{n} E[\sum_{j=1}^{n} (x_j -\bar{x}) \epsilon_j^2] \\ & = 2 \frac{x_i - \bar{x}}{\sum_{j=1}^{n}(x_j - \bar{x})^2} \frac{1}{n} \sigma^2 \sum_{j=1}^{n} (x_j -\bar{x}) \\ & = 0 \\ \therefore E[\hat{\epsilon_i}^2] & = \sigma^2 - \frac{\sigma^2}{n} - \frac{(x_i -\bar{x})^2 \sigma^2}{\sum_{j=1}^{n} (x_j -\bar{x})^2} \\ & = \sigma^2\left(1-\frac{1}{n}- \frac{(x_i -\bar{x})^2}{\sum_{j=1}^{n} (x_j -\bar{x})^2} \right) \end{split} \end{equation}

$\endgroup$
  • 1
    $\begingroup$ My hasty algebra is saying $$ \operatorname{E} \left( \hat \varepsilon_i^2 \right) = \sigma^2\left( 1 - \frac{(x_i-\bar x)^2}{\sum_{j=1}^n (x_j-\bar x)^2} \right).$$ If that is correct, then it's unbiased precisely when $x_i = \bar x$. $\qquad$ $\endgroup$ – Michael Hardy Aug 30 '16 at 2:46
  • $\begingroup$ But I proved differently ... $\endgroup$ – Eric Xu Aug 30 '16 at 3:17
  • $\begingroup$ I'm accustomed to do this by using matrix algebra (the nonlinear part being inversion of a matrix). $\qquad$ $\endgroup$ – Michael Hardy Aug 30 '16 at 3:29
  • $\begingroup$ @MichaelHardy. Can you check my matrix based computations and see if everything is OK? $\endgroup$ – Arin Chaudhuri Aug 30 '16 at 4:50
  • $\begingroup$ ok, I may have missed $1/n$. In this document on page 18 (which is the third page of the pdf file) we find $$ h_{ij} = \frac 1 n + \left. [(x_i-\bar x)(x_j - \bar x)] \middle/ \left[ \sum_{k=1}^n (x_k - \bar x)^2 \right] \right. $$ Here what we need is $1$ minus the diagonal entry of the hat matrix. $\qquad$ $\endgroup$ – Michael Hardy Aug 30 '16 at 17:37
1
$\begingroup$

You have proved (correctly IMHO) that $\texttt{E}(\hat{\epsilon}_i^2) \leq \sigma^2(1 - \frac{1}{n}) < \sigma^2,$ when ${\bf x_i}$'s are not all the same so ${\bf \sum_{i} (x_i - \bar{x})^2 > 0}$, which means $\hat{\epsilon}_i^2$ cannot be unbiased when $x_i$'s are not identical. For consistency you have to make some assumptions about $x_i$'s as $n \to \infty$. If you assume for example $\sum_{i=1}^n(x_i - \bar{x}_n)^2 \to \infty$ (where $\bar{x}_n = \dfrac{\sum_i^n x_i}{n}$). Then clearly keeping $i$ fixed and letting $n \to \infty$ $\texttt{E}(\hat{\epsilon_i}^2) \to \sigma^2.$

Why I think your computation is correct for the case when $x_i$'s are not the same lies below.

Consider the usual linear model setting.

$Y = X\beta + \epsilon$ where $\epsilon \sim N(0,\sigma^2I).$

$\hat{\epsilon} = Y - \hat{Y} = (I - P)Y = (I-P)(Y-X\beta).$ Where $P$ is the projection operator into the column space of $X$.

We have $$ \begin{align}\texttt{E}(\hat{\epsilon}{\hat\epsilon}^T) &= \texttt{E}((I-P)(Y-X\beta)(Y-X\beta)^T(I-P))\\ &=(I-P)\texttt{E}((Y-X\beta)(Y-X\beta)^T)(I-P))\\ &=\sigma^2 (I - P).\end{align}$$

So $\texttt{E}(\hat\epsilon_i^2) = \sigma^2(1 - p_{ii}),$ where $P=(p_{ij}).$

In this particular case $X = \begin{pmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots\\ 1 & x_n \end{pmatrix}$ and $$P = UU^T$$ where $U = \begin{pmatrix}\dfrac{1}{\sqrt{n}} & \dfrac{x_1 - \bar{x}}{\sqrt{\sum_{i=1}^n(x_i-\bar{x})^2}} \\ \dfrac{1}{\sqrt{n}} & \dfrac{x_2 - \bar{x}}{\sqrt{\sum_{i=1}^n(x_i-\bar{x})^2}} \\ \vdots & \vdots & \vdots\\ \dfrac{1}{\sqrt{n}} & \dfrac{x_n - \bar{x}}{\sqrt{\sum_{i=1}^n(x_i-\bar{x})^2}} \end{pmatrix}$ since the columns of $U$ form an orthonormal basis of column space of $X.$

So $p_{ij} = \dfrac{1}{n} + \dfrac{ (x_i - \bar{x})(x_j - \bar{x}) }{\sum_{i=1}^{n}(x_i - \bar{x})^2}$ and $\texttt{E}(\hat\epsilon_i^2) = \sigma^2(1 - p_{ii}) = \sigma^2\left(1 - \dfrac{1}{n} - \dfrac{ (x_i - \bar{x})^2 }{\sum_{i=1}^{n}(x_i - \bar{x})^2}\right).$

However if $x_i$'s are identical. Then $P=\dfrac{{\bf 11}^T}{n}$ ( here ${\bf 1}$ denotes a vector of 1's) and even in this case $\texttt{E}(\hat\epsilon_i^2) = \sigma^2(1-\dfrac{1}{n}) < \sigma^2.$ So $\hat\epsilon_i^2$ is always biased.

$\endgroup$
  • $\begingroup$ But michael has a different answer and I feel the question implies $\hat{\epsilon}^2$ can be unbiased. $\endgroup$ – Eric Xu Aug 30 '16 at 3:55
  • 1
    $\begingroup$ I will present the matrix version of the computation. I think you are correct, I got the same answer using a different method. $\endgroup$ – Arin Chaudhuri Aug 30 '16 at 4:00
  • $\begingroup$ @EricXu : See my matrix based computations. I don't think we can ever have unbiasedness. $\endgroup$ – Arin Chaudhuri Aug 30 '16 at 4:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.