Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Singular value decomposition (SVD) and principal component analysis (PCA) are two eigenvalue methods used to reduce a high-dimensional dataset into fewer dimensions while retaining important information. Articles online say that these methods are 'related' but never specify the exact relation.

What is the intuitive relationship between PCA and SVD? As PCA uses the SVD in its calculation, clearly there is some 'extra' analysis done. What does PCA 'pay attention' to differently than the SVD? What kinds of relationships do each method utilize more in their calculations? Is one method 'blind' to a certain type of data that the other is not?

share|improve this question
1  
SVD and PCA and "total least-squares" (and several other names) are the same thing. It computes the orthogonal transform that decorrelates the variables and keeps the ones with the largest variance. There are two numerical approaches: one by SVD of the (centered) data matrix, and one by Eigen decomposition of this matrix "squared" (covariance). –  Yves Daoust Jun 10 at 8:21

2 Answers 2

up vote 100 down vote accepted

(I assume for the purposes of this answer that the data has been preprocessed to have zero mean.)

Simply put, the PCA viewpoint requires that one compute the eigenvalues and eigenvectors of the covariance matrix, which is the product $\mathbf X\mathbf X^\top$, where $\mathbf X$ is the data matrix. Since the covariance matrix is symmetric, the matrix is diagonalizable, and the eigenvectors can be normalized such that they are orthonormal:

$\mathbf X\mathbf X^\top=\mathbf W\mathbf D\mathbf W^\top$

On the other hand, applying SVD to the data matrix $\mathbf X$ as follows:

$\mathbf X=\mathbf U\mathbf \Sigma\mathbf V^\top$

and attempting to construct the covariance matrix from this decomposition gives

$\begin{align*} \mathbf X\mathbf X^\top&=(\mathbf U\mathbf \Sigma\mathbf V^\top)(\mathbf U\mathbf \Sigma\mathbf V^\top)^\top\\ \mathbf X\mathbf X^\top&=(\mathbf U\mathbf \Sigma\mathbf V^\top)(\mathbf V\mathbf \Sigma\mathbf U^\top) \end{align*}$

and since $\mathbf V$ is an orthogonal matrix ($\mathbf V^\top \mathbf V=\mathbf I$),

$\mathbf X\mathbf X^\top=\mathbf U\mathbf \Sigma^2 \mathbf U^\top$

and the correspondence is easily seen (the square roots of the eigenvalues of $\mathbf X\mathbf X^\top$ are the singular values of $\mathbf X$, etc.)

In fact, using the SVD to perform PCA makes much better sense numerically than forming the covariance matrix to begin with, since the formation of $\mathbf X\mathbf X^\top$ can cause loss of precision. This is detailed in books on numerical linear algebra, but I'll leave you with an example of a matrix that can be stable SVD'd, but forming $\mathbf X\mathbf X^\top$ can be disastrous, the Läuchli matrix:

$\begin{pmatrix}1&1&1\\ \epsilon&0&0\\0&\epsilon&0\\0&0&\epsilon\end{pmatrix}^\top$

where $\epsilon$ is a tiny number.

share|improve this answer
1  
To give a Mathematica example: A=SparseArray[{{i_, 1} -> 1, {i_, j_} /; i + 1 == j :> $MachineEpsilon}, {3, 4}]; and then compare Sqrt[Eigenvalues[a.Transpose[a]]] and SingularValueList[a,Tolerance->0]. –  J. M. Sep 2 '10 at 14:13
2  
If $X$ is the data matrix, isn't the covariance matrix defined by $E[XX^T] - E[X]E[X]^T$, not $XX^T$? And so PCA is equivalent to SVD on the mean centered data matrix. –  JasonMond May 3 '11 at 21:26
2  
@Jason: Ah yes, I was assuming that the data was already "zero mean" for the purposes of this answer. I'll edit in your note later. –  J. M. May 4 '11 at 0:21
2  
I'm not sure about this, but I think the covariance matrix of an $n \times m$ data matrix (with $n$ samples each of size $m$) is $(X X^T) / (n - 1)$, so your example only works if the data matrix is standardized to have unit variance along each column. Somebody please correct me if I'm wrong. –  Jeff Terrell Jan 29 '13 at 21:54
1  
@JeffTerrell Right so, but I think the normalizing factor was left out for the sake of readability. –  fotNelton Feb 13 '13 at 9:18

A tutorial on Principal Component Analysis by Jonathon Shlens is a good tutorial on PCA and its relation to SVD.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.