15
$\begingroup$

I understand that the trace of the projection matrix (also known as the "hat" matrix) X*Inv(X'X)*X' in linear regression is equal to the rank of X. How can we prove that from first principles, i.e. without simply asserting that the trace of a projection matrix always equals its rank?

I am aware of the post Proving: "The trace of an idempotent matrix equals the rank of the matrix", but need an integrated proof.

$\endgroup$
3
  • 2
    $\begingroup$ Does the identity $\text{tr}(AB) = \text{tr}(BA)$ count as being from first principles? $\endgroup$
    – JimmyK4542
    Dec 19, 2015 at 21:58
  • $\begingroup$ I suppose so. One can write that out with indices and see that it holds true. $\endgroup$
    – ClarPaul
    Dec 19, 2015 at 22:28
  • 1
    $\begingroup$ Oh, so, one writes: tr(X*in(X'X)*X')=tr(X'*X*inv(X'X))=tr(I)=dim(X'X) !! Great, thanks! $\endgroup$
    – ClarPaul
    Dec 19, 2015 at 22:38

2 Answers 2

Reset to default
23
$\begingroup$

If $X$ is $n \times m$ with $m \le n$ and has full rank, then $rank (X) = \min(n,m) = m$, and we know $(X^T X)^{-1}$ exists.

By commutativity of the trace operator, we have

$$tr(H) := tr (X (X^T X)^{-1} X^T) = tr (X^T X (X^T X)^{-1} ) = tr[I_m] = m$$

$\endgroup$
4
$\begingroup$

Let $X$ be an $n \times r$ matrix with full rank, i.e. $\text{rank}X = r$ (this is necessary for $(X^TX)^{-1}$ to exist).

Then, using the identity $\text{tr}(AB) = \text{tr}(BA)$, we have

$\text{tr}[X(X^TX)^{-1}X^T] = \text{tr}[X^TX(X^TX)^{-1}] = \text{tr}[I_{r \times r}] = r = \text{rank}X$, as desired.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.