In PCA algorithm, $C$ the covariance matrix is defined as $\frac1{N-1}XX^T$, where $X$ is a matrix of size $d\times n$. Also, the rank of $X$ is $\min(d-1,n)$. (Lets assume)
Now, since $C$ is symmetric and PSD, all its eigenvalues are real and non negative.
We know that number of eigenvalues of a $d \times d$ matrix is $d$.
Now my question(s) are,
(i) how do we know that there are exactly, $r=\min(d,n-1)$ linearly independent eigenvectors corresponding to $r$ non-zero eigenvalues?
(ii) and that the rest $0$ eigenvalues correspond to $(d-r)$, linearly independent eigenvectors.
(iii) and also that even together, all these eigenvectors are linearly independent. ((iv) are they all orthogonal).
I am saying this because in PCA algorithm, $$CV=V\Lambda$$ where $V$ is $d\times d$ eigenvector (at columns) matrix, which is orthogonal, and $\Lambda$ is a $d\times d$ eigenvalue matrix.

Note that to prove the claims, I cannot use SVD, since it is introduced later, and is derived using the facts I wrote above (in some way).


1 Answer 1


Every symmetric matrix is orthogonally diagonalizable.

The easiest way to prove this is using the SVD, but if you don't want to use the SVD, you can start by proving the easier fact that if a matrix has two distinct eigenvalues, then the corresponding eigenvectors are orthogonal.

  • $\begingroup$ Thats exactly what I was looking for. Answers all my queries! Thank you so much! $\endgroup$ Nov 5, 2020 at 21:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.