2
$\begingroup$

How can I prove the variance of residuals in simple linear regression?

Please help me.

$ \operatorname{var}(r_i)=\sigma^2\left[1-\frac{1}{n}-\dfrac{(x_i-\bar{x})^2}{\sum_{l=1}^{n}(x_l-\bar{x})}\right]$

I tried..

using $r_i=y_i-\hat{y_i}$

$\operatorname{var}(r_i)=\operatorname{var}(y_i-\hat{y_i})=\operatorname{var}(y_i-\bar{y})+\operatorname{var}(\hat{\beta_1}(x_i-\bar{x}))-2\operatorname{Cov}((y_i-\bar{y}),\hat{\beta_1}(x_i-\bar{x}))$

How can I go further?

If there's more information needed, please ask me to provide it.

$\endgroup$
2
$\begingroup$

Note that $$\begin{align}\operatorname{Var}(r_i) &=\operatorname{Var}(y_i-\hat{y_i}) \\ &=\operatorname{Var}(y_i)+ \operatorname{Var}(\hat{y_i}) \\ &= \sigma^2 + \operatorname{Var}(\bar y+\hat \beta _1(x_i-\bar x))\\ &= \sigma^2 +\operatorname{Var}(\bar y)+(x_i-\bar x)^2 \cdot\operatorname{Var}(\hat \beta _1)\\&=\sigma^2 +\dfrac{\sigma^2}{n}+\dfrac{\sigma^2 \cdot (x_i-\bar x)^2}{\sum_{i=1}^n(x_i-\bar x)^2}\\ &=\sigma^2 \left[1+\dfrac 1n +\dfrac{(x_i-\bar x)^2}{\sum_{i=1}^n(x_i-\bar x)^2} \right]\end{align}$$

$\endgroup$
  • 2
    $\begingroup$ This incorrectly assumes that $y$ and $\hat{y}$ are uncorrelated. That this is not generally the case should be obvious (if they were uncorrelated, the slope would be 0). $\endgroup$ – Glen_b Sep 11 '14 at 0:39
2
$\begingroup$

I believe the previous answer posted is incorrect, since $y_i$ and $\hat y_i$ are not uncorrelated. I would prove this as follows:

$\begin{align} \text{Cov}(r) &= \text{Cov}(y - Py), \quad P = X(X^TX)^{-1}X^T \\ & = \text{Cov}((I_n-P)y) \\ & = (I_n - P)\ \text{Cov}(y)\ (I_n - P)^T \\ & = (I_n-P)\ \sigma^2 I_n\ (I_n - P)^T \end{align}$

from which we can conclude that $\text{var}(r_i)=\sigma^2 (1 - P_{ii})$. It should be quite simple to confirm that your equation is recovered when you let $X$ be the matrix with a column of $1$'s (to represent $\bar x$) and a second column of the $x_i$'s.

$\endgroup$
  • $\begingroup$ This is not simple linear regression anymore since you are using vectors rather than scalars. $\endgroup$ – Fermat's Student Oct 1 '14 at 7:06
  • $\begingroup$ @Will, that is why I said "let X be the matrix with a column of 1's (to represent x¯) and a second column of the xi's." $\endgroup$ – Brendon Oct 1 '14 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.