Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

The wiki article on eigenvectors offers the following geometrical interpretation:

Each application of the matrix to an arbitrary vector yields a result which will have rotated towards the eigenvector with the largest eigenvalue.

Qn 1: If there is any other geometrical interpretation particularly in the context of a covariance matrix?

The wiki also discusses the difference between left and right eigenvectors.

Qn 2: Do the above geometrical interpretations hold irrespective of whether they are left or right eigenvectors?

share|improve this question
2  
The left eigenvectors of M are the right eigenvectors of M^T, so if you understand the transpose then the geometric interpretations are essentially the same. –  Qiaochu Yuan Jul 29 '10 at 2:11
2  
Principal Component Analysis transforms the data to a basis of eigenvectors with the largest eigenvalues in order to reduce dimension while minimizing information loss. –  Larry Wang Jul 29 '10 at 2:12
    
@Qiaochu: That makes a lot of sense. –  user116 Jul 29 '10 at 2:15
    
@Kaestur: So my understanding of your comment and based on the wiki article is this: The eigenvector associated with the highest eigenvalue is parallel to the major axis of the ellipsoid associated with the covariance matrix. Would that be correct? –  user116 Jul 29 '10 at 2:21
    
@Srikant: Mostly correct. Clustering and such can make it a little complicated. I realized wikipedia does not have a good picture, perhaps this may help. –  Larry Wang Jul 29 '10 at 2:43
show 1 more comment

4 Answers

up vote 10 down vote accepted

Here is a partial answer in the case where M is a real symmetric matrix. This is to ensure, by the real spectral theorem, that M has real eigenvectors with real eigenvalues, so there is a chance for a genuine geometric interpretation which stays in $R^n$.

M acts on the unit sphere in $R^n$ in the following way: it sends the unit sphere $v^T v = 1$ to $v^T (M^T M) v = 1$ . This modified shape is not generally a sphere, but is generally an ellipsoid. The axes of this ellipsoid are the eigenvectors of M, and the sizes of each axis are given by the squares of the corresponding eigenvalues.

share|improve this answer
    
+1 @Qiaochu: I find your method of thinking about "how M modifies the unit sphere" to be very interesting and useful. By the way, why do you say it is a partial answer? Your comment on my second qn + the above response answers my question (at least from my perspective). –  user116 Jul 29 '10 at 2:55
    
I haven't addressed the case where M is allowed to be symmetric or complex. –  Qiaochu Yuan Jul 29 '10 at 3:47
    
Fair enough but I am ok with your answer. –  user116 Jul 29 '10 at 4:24
add comment

If you are interested in covariance matrices, then the eigenvectors of the covariance matrix tell you how to change variables to make your random variables uncorrelated.

Specifically, let $M$ be a covariance matrix of the random variables $X_1,...,X_n$. For simplicity, lets assume that all of these random variables are zero mean. Lets also define $X$ to be the random vector whose $i$"th component is the random variable $X_i$. Let the eigenvectors of $M$ be $v_1, v_2, ..., v_n$. We can assume that these are orthogonal since $M$ is symmetric. Consider the random variables $Y_i$ obtained by taking the dot product of $v_i$ and $X$.

Then the random variables $Y_1, ..., Y_n$ are uncorrelated!

Indeed:

$$ E[ Y_i Y_j] = E[ v_i^T X X^T v_j] = v_i^T M v_j = \lambda_j v_i^T v_j = 0 $$

share|improve this answer
1  
that's what i was looking for. good show! –  Jason S Jul 29 '10 at 12:48
add comment

Instead of giving an answer, let me point out to you this chapter in Cleve Moler's book "Numerical Computing with MATLAB", there is a nice geometric demonstration in MATLAB on how eigenvalues/eigenvectors (as well as singular values/vectors) of an order-2 square matrix are involved in how a circle is transformed into an ellipse after a linear transformation represented by the matrix.

share|improve this answer
    
Current projections indicate tomorrow. :-) –  Asaf Karagila Jul 31 '12 at 0:26
    
Lo and behold, the screenshot. –  Asaf Karagila Aug 1 '12 at 19:05
add comment

Of course! Consider a coordinate transformation of rotation and/or scaling (but not translation):

v = Au

where v and u are vectors, and A is a transformation matrix. Then the eigenvectors, if they have real components, are the axes which are left unrotated (scaling only) by the transformation. (see wikipedia)

A covariance matrix is a symmetric, positive definite matrix, so it has orthonormal eigenvectors, and these form a tuple of axes; I am fairly sure the eigenvectors form a new basis of linear combinations of the input variables where the basis variables are uncorrelated, but I can't remember how to show this.

For example, if w1 = [x;y] is a pair of independent unit-variance zero-mean Gaussian random variables, consider w2 = [u;v] = [1 1; 2 1][x;y] = (x+y,2x+y), so that w1 = [-1 1;2 -1][u;v] = [v-u;2u-v]. Then cov(w2) = [2 3; 3 5]. This has eigenvectors which have sqrt(5) in them, hmmmm...

As for question 2, I'm not sure.

share|improve this answer
1  
But, this seems trivial as it follows from the definition of the eigenvector or is there something deeper I am missing? –  user116 Jul 29 '10 at 1:31
    
@Srikant, could you explain in more detail what sort of answer you are looking for? –  Jonathan Fischoff Jul 29 '10 at 1:33
    
I thought the question was clear. I am looking to see if there is any deeper insight along the lines of the one suggested by the wiki. The geometrical interpretation that repeated applications will rotate a vector towards the eigenvector with the highest eigenvalue is not at all obvious at first sight. I also want to know if there is a difference between left and right eigenvectors with respect to the geometry. Does that help clarify? –  user116 Jul 29 '10 at 1:37
    
I added the section on covariance matrices. forgot how to show what I wanted to show, though. –  Jason S Jul 29 '10 at 2:00
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.