12
$\begingroup$

I understand that the trace of the projection matrix (also known as the "hat" matrix) X*Inv(X'X)*X' in linear regression is equal to the rank of X. How can we prove that from first principles, i.e. without simply asserting that the trace of a projection matrix always equals its rank?

I am aware of the post Proving: "The trace of an idempotent matrix equals the rank of the matrix", but need an integrated proof.

| cite | improve this question | |
$\endgroup$
  • 2
    $\begingroup$ Does the identity $\text{tr}(AB) = \text{tr}(BA)$ count as being from first principles? $\endgroup$ – JimmyK4542 Dec 19 '15 at 21:58
  • $\begingroup$ I suppose so. One can write that out with indices and see that it holds true. $\endgroup$ – clarpaul Dec 19 '15 at 22:28
  • 1
    $\begingroup$ Oh, so, one writes: tr(X*in(X'X)*X')=tr(X'*X*inv(X'X))=tr(I)=dim(X'X) !! Great, thanks! $\endgroup$ – clarpaul Dec 19 '15 at 22:38
17
$\begingroup$

If $X$ is $n \times m$ with $m \le n$ and has full rank, then $rank (X) = \min(n,m) = m$, and we know $(X^T X)^{-1}$ exists.

By commutativity of the trace operator, we have

$$tr(H) := tr (X (X^T X)^{-1} X^T) = tr (X^T X (X^T X)^{-1} ) = tr[I_m] = m$$

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

Let $X$ be an $n \times r$ matrix with full rank, i.e. $\text{rank}X = r$ (this is necessary for $(X^TX)^{-1}$ to exist).

Then, using the identity $\text{tr}(AB) = \text{tr}(BA)$, we have

$\text{tr}[X(X^TX)^{-1}X^T] = \text{tr}[X^TX(X^TX)^{-1}] = \text{tr}[I_{r \times r}] = r = \text{rank}X$, as desired.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.