11
$\begingroup$

The wiki article on eigenvectors offers the following geometrical interpretation:

Each application of the matrix to an arbitrary vector yields a result which will have rotated towards the eigenvector with the largest eigenvalue.

Qn 1: If there is any other geometrical interpretation particularly in the context of a covariance matrix?

The wiki also discusses the difference between left and right eigenvectors.

Qn 2: Do the above geometrical interpretations hold irrespective of whether they are left or right eigenvectors?

$\endgroup$
  • 2
    $\begingroup$ The left eigenvectors of M are the right eigenvectors of M^T, so if you understand the transpose then the geometric interpretations are essentially the same. $\endgroup$ – Qiaochu Yuan Jul 29 '10 at 2:11
  • 2
    $\begingroup$ Principal Component Analysis transforms the data to a basis of eigenvectors with the largest eigenvalues in order to reduce dimension while minimizing information loss. $\endgroup$ – Larry Wang Jul 29 '10 at 2:12
  • $\begingroup$ @Qiaochu: That makes a lot of sense. $\endgroup$ – user116 Jul 29 '10 at 2:15
  • $\begingroup$ @Kaestur: So my understanding of your comment and based on the wiki article is this: The eigenvector associated with the highest eigenvalue is parallel to the major axis of the ellipsoid associated with the covariance matrix. Would that be correct? $\endgroup$ – user116 Jul 29 '10 at 2:21
  • $\begingroup$ @Srikant: Mostly correct. Clustering and such can make it a little complicated. I realized wikipedia does not have a good picture, perhaps this may help. $\endgroup$ – Larry Wang Jul 29 '10 at 2:43
12
$\begingroup$

Here is a partial answer in the case where M is a real symmetric matrix. This is to ensure, by the real spectral theorem, that M has real eigenvectors with real eigenvalues, so there is a chance for a genuine geometric interpretation which stays in $R^n$.

M acts on the unit sphere in $R^n$ in the following way: it sends the unit sphere $v^T v = 1$ to $v^T (M^T M) v = 1$ . This modified shape is not generally a sphere, but is generally an ellipsoid. The axes of this ellipsoid are the eigenvectors of M, and the sizes of each axis are given by the squares of the corresponding eigenvalues.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ +1 @Qiaochu: I find your method of thinking about "how M modifies the unit sphere" to be very interesting and useful. By the way, why do you say it is a partial answer? Your comment on my second qn + the above response answers my question (at least from my perspective). $\endgroup$ – user116 Jul 29 '10 at 2:55
  • 1
    $\begingroup$ I haven't addressed the case where M is allowed to be symmetric or complex. $\endgroup$ – Qiaochu Yuan Jul 29 '10 at 3:47
  • $\begingroup$ Fair enough but I am ok with your answer. $\endgroup$ – user116 Jul 29 '10 at 4:24
6
$\begingroup$

If you are interested in covariance matrices, then the eigenvectors of the covariance matrix tell you how to change variables to make your random variables uncorrelated.

Specifically, let $M$ be a covariance matrix of the random variables $X_1,...,X_n$. For simplicity, lets assume that all of these random variables are zero mean. Lets also define $X$ to be the random vector whose $i$"th component is the random variable $X_i$. Let the eigenvectors of $M$ be $v_1, v_2, ..., v_n$. We can assume that these are orthogonal since $M$ is symmetric. Consider the random variables $Y_i$ obtained by taking the dot product of $v_i$ and $X$.

Then the random variables $Y_1, ..., Y_n$ are uncorrelated!

Indeed:

$$ E[ Y_i Y_j] = E[ v_i^T X X^T v_j] = v_i^T M v_j = \lambda_j v_i^T v_j = 0 $$

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ that's what i was looking for. good show! $\endgroup$ – Jason S Jul 29 '10 at 12:48
6
$\begingroup$

Instead of giving an answer, let me point out to you this chapter in Cleve Moler's book "Numerical Computing with MATLAB", there is a nice geometric demonstration in MATLAB on how eigenvalues/eigenvectors (as well as singular values/vectors) of an order-2 square matrix are involved in how a circle is transformed into an ellipse after a linear transformation represented by the matrix.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

Of course! Consider a coordinate transformation of rotation and/or scaling (but not translation):

v = Au

where v and u are vectors, and A is a transformation matrix. Then the eigenvectors, if they have real components, are the axes which are left unrotated (scaling only) by the transformation. (see wikipedia)

A covariance matrix is a symmetric, positive definite matrix, so it has orthonormal eigenvectors, and these form a tuple of axes; I am fairly sure the eigenvectors form a new basis of linear combinations of the input variables where the basis variables are uncorrelated, but I can't remember how to show this.

For example, if w1 = [x;y] is a pair of independent unit-variance zero-mean Gaussian random variables, consider w2 = [u;v] = [1 1; 2 1][x;y] = (x+y,2x+y), so that w1 = [-1 1;2 -1][u;v] = [v-u;2u-v]. Then cov(w2) = [2 3; 3 5]. This has eigenvectors which have sqrt(5) in them, hmmmm...

As for question 2, I'm not sure.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ But, this seems trivial as it follows from the definition of the eigenvector or is there something deeper I am missing? $\endgroup$ – user116 Jul 29 '10 at 1:31
  • $\begingroup$ @Srikant, could you explain in more detail what sort of answer you are looking for? $\endgroup$ – Jonathan Fischoff Jul 29 '10 at 1:33
  • $\begingroup$ I thought the question was clear. I am looking to see if there is any deeper insight along the lines of the one suggested by the wiki. The geometrical interpretation that repeated applications will rotate a vector towards the eigenvector with the highest eigenvalue is not at all obvious at first sight. I also want to know if there is a difference between left and right eigenvectors with respect to the geometry. Does that help clarify? $\endgroup$ – user116 Jul 29 '10 at 1:37
  • $\begingroup$ I added the section on covariance matrices. forgot how to show what I wanted to show, though. $\endgroup$ – Jason S Jul 29 '10 at 2:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy