# Is there a geometrical interpretation to the notion of eigenvector and eigenvalues?

The wiki article on eigenvectors offers the following geometrical interpretation:

Each application of the matrix to an arbitrary vector yields a result which will have rotated towards the eigenvector with the largest eigenvalue.

Qn 1: If there is any other geometrical interpretation particularly in the context of a covariance matrix?

The wiki also discusses the difference between left and right eigenvectors.

Qn 2: Do the above geometrical interpretations hold irrespective of whether they are left or right eigenvectors?

-
The left eigenvectors of M are the right eigenvectors of M^T, so if you understand the transpose then the geometric interpretations are essentially the same. –  Qiaochu Yuan Jul 29 '10 at 2:11
Principal Component Analysis transforms the data to a basis of eigenvectors with the largest eigenvalues in order to reduce dimension while minimizing information loss. –  Larry Wang Jul 29 '10 at 2:12
@Qiaochu: That makes a lot of sense. –  user116 Jul 29 '10 at 2:15
@Kaestur: So my understanding of your comment and based on the wiki article is this: The eigenvector associated with the highest eigenvalue is parallel to the major axis of the ellipsoid associated with the covariance matrix. Would that be correct? –  user116 Jul 29 '10 at 2:21
@Srikant: Mostly correct. Clustering and such can make it a little complicated. I realized wikipedia does not have a good picture, perhaps this may help. –  Larry Wang Jul 29 '10 at 2:43

Here is a partial answer in the case where M is a real symmetric matrix. This is to ensure, by the real spectral theorem, that M has real eigenvectors with real eigenvalues, so there is a chance for a genuine geometric interpretation which stays in $R^n$.

M acts on the unit sphere in $R^n$ in the following way: it sends the unit sphere $v^T v = 1$ to $v^T (M^T M) v = 1$ . This modified shape is not generally a sphere, but is generally an ellipsoid. The axes of this ellipsoid are the eigenvectors of M, and the sizes of each axis are given by the squares of the corresponding eigenvalues.

-
+1 @Qiaochu: I find your method of thinking about "how M modifies the unit sphere" to be very interesting and useful. By the way, why do you say it is a partial answer? Your comment on my second qn + the above response answers my question (at least from my perspective). –  user116 Jul 29 '10 at 2:55
I haven't addressed the case where M is allowed to be symmetric or complex. –  Qiaochu Yuan Jul 29 '10 at 3:47
Fair enough but I am ok with your answer. –  user116 Jul 29 '10 at 4:24

If you are interested in covariance matrices, then the eigenvectors of the covariance matrix tell you how to change variables to make your random variables uncorrelated.

Specifically, let $M$ be a covariance matrix of the random variables $X_1,...,X_n$. For simplicity, lets assume that all of these random variables are zero mean. Lets also define $X$ to be the random vector whose $i$"th component is the random variable $X_i$. Let the eigenvectors of $M$ be $v_1, v_2, ..., v_n$. We can assume that these are orthogonal since $M$ is symmetric. Consider the random variables $Y_i$ obtained by taking the dot product of $v_i$ and $X$.

Then the random variables $Y_1, ..., Y_n$ are uncorrelated!

Indeed:

$$E[ Y_i Y_j] = E[ v_i^T X X^T v_j] = v_i^T M v_j = \lambda_j v_i^T v_j = 0$$

-
that's what i was looking for. good show! –  Jason S Jul 29 '10 at 12:48

Instead of giving an answer, let me point out to you this chapter in Cleve Moler's book "Numerical Computing with MATLAB", there is a nice geometric demonstration in MATLAB on how eigenvalues/eigenvectors (as well as singular values/vectors) of an order-2 square matrix are involved in how a circle is transformed into an ellipse after a linear transformation represented by the matrix.

-
Current projections indicate tomorrow. :-) –  Asaf Karagila Jul 31 '12 at 0:26
Lo and behold, the screenshot. –  Asaf Karagila Aug 1 '12 at 19:05

Of course! Consider a coordinate transformation of rotation and/or scaling (but not translation):

v = Au


where v and u are vectors, and A is a transformation matrix. Then the eigenvectors, if they have real components, are the axes which are left unrotated (scaling only) by the transformation. (see wikipedia)

A covariance matrix is a symmetric, positive definite matrix, so it has orthonormal eigenvectors, and these form a tuple of axes; I am fairly sure the eigenvectors form a new basis of linear combinations of the input variables where the basis variables are uncorrelated, but I can't remember how to show this.

For example, if w1 = [x;y] is a pair of independent unit-variance zero-mean Gaussian random variables, consider w2 = [u;v] = [1 1; 2 1][x;y] = (x+y,2x+y), so that w1 = [-1 1;2 -1][u;v] = [v-u;2u-v]. Then cov(w2) = [2 3; 3 5]. This has eigenvectors which have sqrt(5) in them, hmmmm...

As for question 2, I'm not sure.

-
But, this seems trivial as it follows from the definition of the eigenvector or is there something deeper I am missing? –  user116 Jul 29 '10 at 1:31
@Srikant, could you explain in more detail what sort of answer you are looking for? –  Jonathan Fischoff Jul 29 '10 at 1:33
I thought the question was clear. I am looking to see if there is any deeper insight along the lines of the one suggested by the wiki. The geometrical interpretation that repeated applications will rotate a vector towards the eigenvector with the highest eigenvalue is not at all obvious at first sight. I also want to know if there is a difference between left and right eigenvectors with respect to the geometry. Does that help clarify? –  user116 Jul 29 '10 at 1:37
I added the section on covariance matrices. forgot how to show what I wanted to show, though. –  Jason S Jul 29 '10 at 2:00