I am confused at the variance of residual in regression (matrix form)

Given $Y = X*\beta \hat{}+ \epsilon$

Y is a nx1 predicted vector

X is a nxp matrix

$\beta\hat{}$ is a px1 coefficient vector = $(X^TX)^{-1}X^TY$

$\epsilon$ is a nx1 residual vector

Then $var(\epsilon)=var(Y-X\beta\hat{}$) = var($Y$ - $X(X^TX)^{-1}X^TY$) (1)

Let projection matrix $P$ = $X(X^TX)^{-1}X^T$ . It is idempotent $(P^n=P)$

(1) becomes $var(Y - PY)$ (2)

Method 1: (2) = $var ((I-P)Y) = (I-P)Var(Y)(I-P)^T = \sigma^2(I-P)$

Method 2: (2) = $var(Y)+var(PY) = \sigma^2I + P\sigma^2P^T = \sigma^2(I+P)$

I expect that both methods should result in the same formula but they did not. I am not sure what is missed here


1 Answer 1


The second formula you have assumed independence (or at least zero covariance) of $Y$ and $PY$, that does not hold.

See here, you forgot two important terms.

$$var (Y - PY) = var (Y) + var(PY) + cov(Y, -PY) + cov(-PY, Y) $$

  • $\begingroup$ Yes, thank you for pointing it out. After plugging in these two terms, now I could find the result similar to the first formula. $\endgroup$ Mar 5, 2021 at 18:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.