Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

If I have a covariance matrix for a data set and I multiply it times one of it's eigenvectors. Let's say the eigenvector with the highest eigenvalue. The result is the eigenvector or a scaled version of the eigenvector.

What does this really tell me? Why is this the principal component? What property makes it a principal component? Geometrically, I understand that the principal component (eigenvector) will be sloped at the general slope of the data (loosely speaking). Again, can someone help understand why this happens?

share|improve this question

2 Answers 2

up vote 31 down vote accepted

Short answer: The eigenvector with the largest eigenvalue is the direction along which the data set has the maximum variance. Meditate upon this.

Long answer: Let's say you want to reduce the dimensionality of your data set, say down to just one dimension. In general, this means picking a unit vector $u$, and replacing each data point, $x_i$, with its projection along this vector, $u^T x_i$. Of course, you should choose $u$ so that you retain as much of the variation of the data points as possible: if your data points lay along a line and you picked $u$ orthogonal to that line, all the data points would project onto the same value, and you would lose almost all the information in the data set! So you would like to maximize the variance of the new data values $u^T x_i$. It's not hard to show that if the covariance matrix of the original data points $x_i$ was $\Sigma$, the variance of the new data points is just $u^T \Sigma u$. As $\Sigma$ is symmetric, the unit vector $u$ which maximizes $u^T \Sigma u$ is nothing but the eigenvector with the largest eigenvalue.

If you want to retain more than one dimension of your data set, in principle what you can do is first find the largest principal component, call it $u_1$, then subtract that out from all the data points to get a "flattened" data set that has no variance along $u_1$. Find the principal component of this flattened data set, call it $u_2$. If you stopped here, $u_1$ and $u_2$ would be a basis of the two-dimensional subspace which retains the most variance of the original data; or, you can repeat the process and get as many dimensions as you want. As it turns out, all the vectors $u_1, u_2, \ldots$ you get from this process are just the eigenvectors of $\Sigma$ in decreasing order of eigenvalue. That's why these are the principal components of the data set.

share|improve this answer
2  
Phenomenal answer. Thank you. –  Ryan Mar 28 '11 at 11:55
    
Great explanation, but aren't $u$ and $x$ both column vectors? If this is the case, shouldn't $u^Tx$ be a number, i.e., a 1x1 vector? Why do you call the inner product a vector? I feel the projected vector should be $u^Txu$ instead. Please kindly correct me should I be wrong. Thanks a lot! –  Farticle Pilter 2 days ago
1  
@Farticle Pilter: Yes, $u^Tx$ is a scalar (note that I don't call it a vector). You replace each point $x$ with the number $u^Tx$, turning an $n$-dimensional data set into a $1$-dimensional data set. –  Rahul 2 days ago

Some informal explanation:

Covariance matrix $C_y$ (it is symmetric) encodes the correlations between variables of a vector. In general a covariance matrix is non-diagonal (i.e. have non zero correlations with respect to different variables).

But it's interesting to ask, is it possible to diagonalize the covariance matrix by changing basis of the vector?. In this case there will be no (i.e. zero) correlations between different variables of the vector.

Diagonalization of this symmetric matrix is possible with eigen value decomposition. You may read 'A Tutorial on Principal Component Analysis'by Jonathon Shlens to get a good understanding. (pages 6-7)

share|improve this answer
    
"Diagonalizing the covariance matrix" part is insightful! Thank you! –  Farticle Pilter 2 days ago

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.