It is known that when a matrix is symmetric we can get its SVD $A = U \Sigma V^T$.

So the problem is reduced to compute the eigen-decomposition like

$A = W \Lambda W^T $.

The left singular vectors $u_i$ are $w_i$ and the right singular vectors $v_i$ are $\text{sign}(\lambda_i) w_i$. The singular values $\sigma_i$ are the magnitude of the eigen values $\lambda_i$.

So, if I compute $A= W \Lambda W^T $, how can I do PCA, as far as I get this I would take $\sigma_i = \sqrt{\lambda_i}$ to get eigenvalues, Is this correct? What else can you say about this?


For any matrix A you can get SVD: $A=U\Sigma{}V^T$. Here $U$ is eigenvector of $AA^T$ and $V$ is eigenvector of $A^TA$. You have $AA^T=U\Lambda{}U^T$ Then each element of $\Lambda$: $\lambda_i=\sigma_i^2$. You can simply prove by $AA^T=U\Sigma{}V^TV\Sigma^TU^T=U\Sigma{}^2U^T$.

If $A$ is your data matrix, each column is a data point, $A$ is not necessarily square, but $AA^T$ is square. If you preprocess your $A$ by $A=A-repmat(mean(A,2),1,size(A,2))$; then your $AA^T$ is the covariant matrix. Because $AA^T$ is positive semidefinite, all the eigenvalues are positive or zero. Take the eigenvectors corresponding to the largest eigenvalues, then you get the PCA projection matrix. Computing the eigenvectors of $AA^T$ is the same as computing the SVD of $A$, so you can just compute the SVD of $A=U\Sigma{}V^T$, and take the first p columns of U as your projection matrix. Project your data $A$ to $U_{(:,1:p)}^TA$.

  • $\begingroup$ if i use correlation matrix of my data, let's say I convert a huge matrix to a $d \times d$ correlation matrix $\rho$, where $d$ is the dimension of the original matrix. If I do SVD of $\rho$ it would be equivalent to $\rho= \mathbf X \mathbf X^{\top}=\mathbf U \mathbf \Sigma^{2} \mathbf U^{\top}$ IS THAT CORRECT? $\endgroup$ – edgarmtze Nov 27 '12 at 6:11
  • $\begingroup$ Yes, because your $\rho$ is symmetric, the SVD is also symmetric. $\endgroup$ – Min Lin Nov 27 '12 at 6:16
  • $\begingroup$ one last question... What would be the relation of covariant matrix (you described) and a $d \times d$ correlation matrix when applying SVD to both of them? $\endgroup$ – edgarmtze Nov 27 '12 at 6:18
  • 1
    $\begingroup$ Covariance matrix is more accurate as PCA, you can imagine dots cloud on a 2D plane with x y span within 1, if the center of the dots is (0,0), then correlation matrix is same as covariance matrix. The first principle component is in the direction that the dots has max variance. But if the dots cloud centered at (100,100), if you don't subtract the mean, then the first principle component is close to (100,100), the second principle component would be close to the first principle component in the de-meaned version. it is only close to, but not exactly the same. You can do some simulation to see $\endgroup$ – Min Lin Nov 27 '12 at 6:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.