0
$\begingroup$

It is known that when a matrix is symmetric we can get its SVD $A = U \Sigma V^T$.

So the problem is reduced to compute the eigen-decomposition like

$A = W \Lambda W^T $.

The left singular vectors $u_i$ are $w_i$ and the right singular vectors $v_i$ are $\text{sign}(\lambda_i) w_i$. The singular values $\sigma_i$ are the magnitude of the eigen values $\lambda_i$.

So, if I compute $A= W \Lambda W^T $, how can I do PCA, as far as I get this I would take $\sigma_i = \sqrt{\lambda_i}$ to get eigenvalues, Is this correct? What else can you say about this?

$\endgroup$

1 Answer 1

2
$\begingroup$

For any matrix A you can get SVD: $A=U\Sigma{}V^T$. Here $U$ is eigenvector of $AA^T$ and $V$ is eigenvector of $A^TA$. You have $AA^T=U\Lambda{}U^T$ Then each element of $\Lambda$: $\lambda_i=\sigma_i^2$. You can simply prove by $AA^T=U\Sigma{}V^TV\Sigma^TU^T=U\Sigma{}^2U^T$.

If $A$ is your data matrix, each column is a data point, $A$ is not necessarily square, but $AA^T$ is square. If you preprocess your $A$ by $A=A-repmat(mean(A,2),1,size(A,2))$; then your $AA^T$ is the covariant matrix. Because $AA^T$ is positive semidefinite, all the eigenvalues are positive or zero. Take the eigenvectors corresponding to the largest eigenvalues, then you get the PCA projection matrix. Computing the eigenvectors of $AA^T$ is the same as computing the SVD of $A$, so you can just compute the SVD of $A=U\Sigma{}V^T$, and take the first p columns of U as your projection matrix. Project your data $A$ to $U_{(:,1:p)}^TA$.

$\endgroup$
4
  • $\begingroup$ if i use correlation matrix of my data, let's say I convert a huge matrix to a $d \times d$ correlation matrix $\rho$, where $d$ is the dimension of the original matrix. If I do SVD of $\rho$ it would be equivalent to $\rho= \mathbf X \mathbf X^{\top}=\mathbf U \mathbf \Sigma^{2} \mathbf U^{\top}$ IS THAT CORRECT? $\endgroup$
    – edgarmtze
    Nov 27, 2012 at 6:11
  • $\begingroup$ Yes, because your $\rho$ is symmetric, the SVD is also symmetric. $\endgroup$
    – Min Lin
    Nov 27, 2012 at 6:16
  • $\begingroup$ one last question... What would be the relation of covariant matrix (you described) and a $d \times d$ correlation matrix when applying SVD to both of them? $\endgroup$
    – edgarmtze
    Nov 27, 2012 at 6:18
  • 1
    $\begingroup$ Covariance matrix is more accurate as PCA, you can imagine dots cloud on a 2D plane with x y span within 1, if the center of the dots is (0,0), then correlation matrix is same as covariance matrix. The first principle component is in the direction that the dots has max variance. But if the dots cloud centered at (100,100), if you don't subtract the mean, then the first principle component is close to (100,100), the second principle component would be close to the first principle component in the de-meaned version. it is only close to, but not exactly the same. You can do some simulation to see $\endgroup$
    – Min Lin
    Nov 27, 2012 at 6:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .