0
$\begingroup$

We have

$$y_i = \beta_0 + \beta_1x_i + \epsilon_i$$ and $$ \hat{y_i} = \hat{\beta_0} + \hat{\beta_1}x_i $$

where $\epsilon_i \sim N(0, \sigma^2)$.

Let $$e_i = y_i - \hat{y_i} $$

I showed that

$$E(e_i) = E(\beta_0 + \beta_1x_i + \epsilon_i - \hat{\beta_0} - \hat{\beta_1}x_i) = \beta_0 + \beta_1x_i - \beta_0 - \beta_1x_i = 0$$

I want to show further that $V(e_i) = \sigma^2$ but I am facing problem in proving this

\begin{align} E(e_i^2) &= E((y_i - \hat{y_i})^2 )= E(y_i^2) + E(\hat{y_i}^2) - 2E(y_i\hat{y_i}) \\&= V(y_i) + E(y_i)^2 + V(\hat{y_i}) + E(\hat{y_i})^2 - 2E((\beta_0 + \beta_1x_i + \epsilon_i)(\hat{\beta_0} + \hat{\beta_1}x_i)) \end{align}

I can see from here that I will get stuck calculating $E(\epsilon_i \hat{\beta_0})$ or $E(\epsilon_i \hat{\beta_1})$.

$\endgroup$

2 Answers 2

2
$\begingroup$

The answer is $$ \operatorname{Var}(e_i) = \sigma^2\left(1-\frac1n-\frac{(x_i-\bar x)^2}{\text{SSX}}\right), $$ where SSX is shorthand for $\sum(x_i-\bar x)^2$.

The derivation is quite involved. Here is one approach. We require the formula for the variance of the difference of two random variables: $$ \operatorname{Var}(A-B)=\operatorname{Var}(A) + \operatorname{Var}(B) - 2\operatorname{Cov}(A,B).\tag{*} $$

  1. Write the $i$th residual in the form $$ e_i:= y_i-\hat y_i = (\epsilon_i-\bar\epsilon)-(\hat\beta_1-\beta_1)(x_i-\bar x)\tag1$$ by plugging in the definitions for $y_i$ and $\hat y_i$ into $y_i-\hat y_i$, and then substituting $\hat \beta_0:=\bar y - \hat\beta_1 \bar x$.

  2. Applying (*) to (1), the desired variance is $$\operatorname{Var}(e_i) = \operatorname{Var}(\epsilon_i-\bar \epsilon) + (x_i-\bar x)^2\operatorname{Var}(\hat\beta_1-\beta_1)-2(x_i-\bar x)\operatorname{Cov}(\epsilon_i-\bar\epsilon, \hat\beta_1-\beta_1).\tag2$$

  3. Using (*), calculate $$\operatorname{Var}(\epsilon_i-\bar\epsilon)=\operatorname{Var}(\epsilon_i) + \operatorname{Var}(\bar\epsilon) - 2\operatorname{Cov}(\epsilon_i,\bar\epsilon)=\sigma^2\left(1-\frac1n\right).\tag3$$ The tricky calculation is $\operatorname{Cov}(\epsilon_i,\bar\epsilon)$, which requires you to observe that $\epsilon_i$ is independent of $\epsilon_k$ when $k\ne i$.

  4. The variance of $\hat\beta_1-\beta_1$ is well known to be $$\operatorname{Var}(\hat\beta_1-\beta_1)=\frac{\sigma^2}{\text{SSX}}.\tag4$$

  5. The covariance in (2) reduces to $E(\epsilon_i-\bar\epsilon)(\hat\beta_1-\beta_1)$, since $E(\hat\beta_1)=\beta_1$. Substitute the formula $$\hat\beta_1-\beta_1=\frac{\sum_k(x_k-\bar x)(\epsilon_k-\bar\epsilon)}{\text{SSX}}\tag 5$$ to obtain $$(\epsilon_i-\bar\epsilon)(\hat\beta_1-\beta_1)=\frac{\sum_k(x_k-\bar x)(\epsilon_i-\bar\epsilon)(\epsilon_k-\bar\epsilon)}{\text{SSX}}.\tag6 $$ Break up the sum in (6) into $\sum_{k=i} + \sum_{k\ne i}$ and take expectations. The answer will be $$\operatorname{Cov}(\epsilon_i-\bar\epsilon, \hat\beta_1-\beta_1)=E(\epsilon_i-\bar\epsilon)(\hat\beta_1-\beta_1)=\frac{(x_i-\bar x)\sigma^2}{\text{SSX}}.\tag 7$$

$\endgroup$
5
  • $\begingroup$ I will go over the calculations but it was given to me already that variance of $e_i$ is $\sigma^2$ (which I have mentioned too). However, it was not from a reliable source and might be wrong. I will go over your calculations and accept your answer soon. Thank you so much. $\endgroup$
    – chesslad
    Commented Jan 24, 2021 at 9:19
  • $\begingroup$ Is there a factor of 2 missing in the covariance term? $\endgroup$
    – Smokey
    Commented Dec 19, 2022 at 6:55
  • $\begingroup$ @Smokey Which equation # are you referring to? $\endgroup$
    – grand_chat
    Commented Dec 19, 2022 at 17:47
  • $\begingroup$ In the final result where we have the term of 2* covariance(A, B) (in Equation *). However, Equation (7) calculates covariance(A,B) $\endgroup$
    – Smokey
    Commented Dec 19, 2022 at 18:44
  • $\begingroup$ @Smokey The desired variance is $\operatorname{Var}(e_i)$, which is written out in (2) using (*). Equations (3), (4), (7) evaluate the required components, which are then substituted into (2). $\endgroup$
    – grand_chat
    Commented Dec 20, 2022 at 17:19
1
$\begingroup$

From your third line, you have assumed that $\epsilon_i \sim N(0, \sigma^2)$. We can see that the distribution of $\epsilon_i$ do not depend on what sample you get and also whatever estimator you come up with. With the independency, check related theorem and I think you can proceed easily.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .