I'm studying Principle Component Analysis (PCA), and came across this post. In which it's written that diagonalization of co-variance matrix ($C$) can be given by $$C = VLV^T$$

But as per difference between SVD and Diagonalization and this post , it's clear that diagonalization of any matrix can be given by: $$C = VLV^{-1}$$

So why the definition of SVD and diagonalization is same here?

  • $\begingroup$ Is the matrix that you're diagonalizing / decomposing symmetric? $\endgroup$
    – user563147
    Jul 21 '18 at 5:13
  • $\begingroup$ Yes, Co-variance matrix is always symmetric. $\endgroup$
    – Kaushal28
    Jul 21 '18 at 5:15
  • $\begingroup$ Then your singular values are exactly the eigenvalues - try to verify this on your own. $\endgroup$
    – user563147
    Jul 21 '18 at 5:17
  • $\begingroup$ I can tell you that a diagonalization of any matrix can be given by the formula you give with $V$ and $V^-1$. As a corolary of the spectral theorem for autoadjoints lineal transformations it is proved that any symmetric matrix is diagonalizable and what is more $V$ can be obtained such that $V^{-1} = V^t$. What I can not find is where is the definition of diagonalization and SVD in the link you post (the last here). $\endgroup$
    – HFKy
    Jul 21 '18 at 5:22
  • $\begingroup$ your singular values are the square root of the eigenvalues $\endgroup$
    – user3417
    Jul 21 '18 at 5:25

The SVD is a generalization of the eigendecomposition. The SVD is the following.

Suppose $ A \in \mathbb{C}^{m \times n}$


$$A = U \Sigma V^{T} $$

where $U,V^{T}$ are orthogonal matrices and $\Sigma $ is a diagonal matrix of singular values. The connection comes here when forming the covariance matrix

$$AA^{T} = (U \Sigma V^{T}) (U \Sigma V^{T})^{T} $$ $$AA^{T} = (U \Sigma V^{T}) (V \Sigma^{T}U^{T})$$ $$ AA^{T} = U \Sigma V^{T} V \Sigma^{T} U^{T}$$

Now $VV^{T} =V^{T}V = I $ $$ AA^{T} = U \Sigma \Sigma^{T} U^{T} $$ Also $ \Sigma^{T} = \Sigma $ $$ AA^{T} = U\Sigma^{2} U^{T}$$ Now we have $ \Sigma^{2} = \Lambda $ $$ AA^{T} = U \Lambda U^{T}$$

The actual way you compute the SVD is pretty similar to the eigendecomp.

In respect to the PCA, it is telling you specifically in the answer you have take the covariance matrix and normalize it (centering). Then it only take the left singular vectors and singular values I believe while truncating it.

A truncated SVD is like this.

$$A_{k} = U_{k}\Sigma_{k} V_{k}^{T} $$

this means the following $$A_{k} = \sum_{i=1}^{k} \sigma_{i} u_{i} v_{i}^{T} $$

So you actually read that they aren't the same. It uses the SVD in forming because it is simpler. The last part states

The product $$ U_{k}\Sigma_{k}$$

gives us a reduction in the dimensionality which contains the first k principal components here we then multiply be the principal axis

$$X_{k} = U_{k}\Sigma_{k} V_{k}^{T} $$

This is commonly referred to as a truncated SVD.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.