Let $Y=X\beta+\epsilon$, where $Y$ is an $n$ by $1$ vector, $X$ is an $n$ by $p$ matrix with full rank and $\epsilon$ is an $n$ by 1 vector of random errors independently and normally distribution with mean vector $0$ and variance-covariance matrix $\Sigma=\sigma^2 I$, with $0$ being an $n$ by $1$ vector of zeros and $I$ being the $n$ by $n$ identity matrix. Prove the least square estimator $\hat{\beta}$ and $Y-X\hat{\beta}$ are independent vectors.

I already get $\hat{\beta}=(X^TX)^{-1}X^{T}Y$. But I tried to prove independence, but I cannot.

  • 2
    $\begingroup$ Hint: show that $\hat{\beta}$ and $Y-X\hat{\beta}$ are jointly normal random vectors. Since the two are uncorrelated, they must also be independent. $\endgroup$ – Dilip Sarwate Oct 19 '14 at 4:34
  • $\begingroup$ Thanks. I am trying to prove $E(\hat{\beta} (Y-X\hat{\beta})^T)=0$. I substitute $\hat{\beta}$, and get $E[(X^{T}X)^{-1} X^{T} Y Y^{T}-(X^{T}X)^{-1} X^{T}Y Y^{T}X(X^{T}X)^{-1}X^{T}]$. But I do not how to proceed. $\endgroup$ – 81235 Oct 19 '14 at 5:44
  • $\begingroup$ I also get $E(\hat{\beta}(Y-X\hat{\beta})^{T})=(X^{T}X)^{-1}X^{T}E(Y Y^{T})-(X^{T}X)^{-1}X^{T}E(Y Y^{T})(X^{T}X)^{-1}X^{T}$. $\endgroup$ – 81235 Oct 19 '14 at 5:58
  • $\begingroup$ The term "independence" is ambiguous as it is used for "statistical independence" but has another meaning in linear algebra. What do you mean here? Note that the vectors $\beta$ and $Y-X\beta$ do not have the same dimension. $\endgroup$ – Bertrand Feb 23 '19 at 14:02

With $\hat\beta:=(X^TX)^{-1}X^TY$ as the vector of estimated parameters and $r:=Y-X\hat\beta$ the vector of residuals, introduce the hat matrix $ H:=X(X^TX)^{-1}X^T$. Check that $X\hat\beta=HY$ and $X=HX$, so $$r:=Y-X\hat\beta=Y-HY=(I-H)Y=(I-H)(X\beta +\epsilon)=(I-H)\epsilon.\tag1 $$ In particular $E( r)=(I-H)E(\epsilon)=0$, so to show $r$ and $\hat\beta$ are uncorrelated it's enough to show $E(r\hat\beta^T)=0$. Indeed: $$ \begin{align} E(r\hat\beta^T)&=E\left(\left[(I-H)\epsilon\right]\left[(X^TX)^{-1}X^TY\right]^T\right)\\ &\stackrel{(a)}=E\left((I-H)\epsilon Y^TX(X^TX)^{-1}\right)\\ &\stackrel{(b)}=(I-H)E(\epsilon Y^T) X(X^TX)^{-1}.\tag2 \end{align} $$ In (a) we use the fact that $(X^TX)^{-1}$ is symmetric, so is its own transpose; in (b) we see that everything except $\epsilon Y^T$ is non-stochastic. The expectation of $\epsilon Y^T$ is $$E(\epsilon Y^T)=E(\epsilon(\epsilon^T+\beta^TX^T)=E(\epsilon\epsilon^T)=:\Sigma=\sigma^2I,\tag3$$ so the RHS of (2) equals $\sigma^2\underbrace{(I-H)X}_{0}(X^TX)^{-1} = 0$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.