58
$\begingroup$

I have a very simple question that can be stated without proof. Are all eigenvectors, of any matrix, always orthogonal? I am trying to understand Principal components and it is cruucial for me to see the basis of eigenvectors.

$\endgroup$
4
  • 8
    $\begingroup$ No. Take any any non-orthogonal basis $(v_1,\dots,v_n)$ and define a linear map $A$ on this basis by sending each $v_i$ to $iv_i$. The eigenspaces are the $n$ lines generated by the $v_i$, and these are by construction not ortgogonal. $\endgroup$ – Olivier Bégassat May 8 '12 at 13:38
  • 13
    $\begingroup$ Eigenvectors corresponding to different eigenvalues will be orthogonal if the matrix is symmetric. This is part of the real spectral theorem. $\endgroup$ – Dylan Moreland May 8 '12 at 13:38
  • 1
    $\begingroup$ The case @Dylan is describing will apply to your study of principal components, since the underlying matrices are symmetric... $\endgroup$ – J. M. ain't a mathematician May 8 '12 at 14:50
  • $\begingroup$ @PL this is clearly not a duplicate of that question - this question is about the general case when $M$ is not necessarily a real symmetric matrix. $\endgroup$ – KReiser Jul 26 '20 at 1:55
28
$\begingroup$

Fix two linearly independent vectors $u$ and $v$ in $\mathbb{R}^2$, define $Tu=u$ and $Tv=2v$. Then extend linearly $T$ to a map from $\mathbb{R}^n$ to itself. The eigenvectors of $T$ are $u$ and $v$ (or any multiple). Of course, $u$ need not be perpendicular to $v$.

$\endgroup$
6
  • $\begingroup$ Thank you very much for your answers. Could anyone state whether they are orthogonal in PCA case? $\endgroup$ – Bober02 May 8 '12 at 13:45
  • 2
    $\begingroup$ For PCA, things can always be set up such that the eigenvectors are orthogonal. On the other hand, I would recommend looking at PCA as a singular value decomposition instead of as an eigendecomposition. It's been discussed here on math.SE a number of times; search around. $\endgroup$ – J. M. ain't a mathematician May 8 '12 at 14:48
  • $\begingroup$ What do you mean by "setting up"? Is there some common technique to achive a singular matrix? $\endgroup$ – Bober02 May 9 '12 at 8:12
  • 3
    $\begingroup$ My understanding based on en.wikipedia.org/wiki/Principal_component_analysis is that a singular value decomposition (SVD) of $X$ results in three matrices, where the first of them is composed of the eigenvectors of $X^T X$, and is used for PCA. This matrix is always orthogonal. $\endgroup$ – Uri Jun 14 '13 at 13:04
  • $\begingroup$ Beautiful answer. $\endgroup$ – Don Larynx Nov 13 '13 at 15:14
35
$\begingroup$

In general, for any matrix, the eigenvectors are NOT always orthogonal. But for a special type of matrix, symmetric matrix, the eigenvalues are always real and the corresponding eigenvectors are always orthogonal.

For any matrix M with n rows and m columns, M multiplies with its transpose, either M*M' or M'M, results in a symmetric matrix, so for this symmetric matrix, the eigenvectors are always orthogonal.

In the application of PCA, a dataset of n samples with m features is usually represented in a n* m matrix D. The variance and covariance among those m features can be represented by a m*m matrix D'*D, which is symmetric (numbers on the diagonal represent the variance of each single feature, and the number on row i column j represents the covariance between feature i and j). The PCA is applied on this symmetric matrix, so the eigenvectors are guaranteed to be orthogonal.

$\endgroup$
4
$\begingroup$

In the context of PCA: it is usually applied to a positive semi-definite matrix, such as a matrix cross product, $X ' X$, or a covariance or correlation matrix.

In this PSD case, all eigenvalues, $\lambda_i \ge 0$ and if $\lambda_i \ne \lambda_j$, then the corresponding eivenvectors are orthogonal. If $\lambda_i = \lambda_j$ then any two orthogonal vectors serve as eigenvectors for that subspace.

$\endgroup$
1
$\begingroup$

Not necessarily all orthogonal. However two eigenvectors corresponding to different eigenvalues are orthogonal.

e.g Let $X_1$ and $X_2$ be two eigenvectors of a matrix $A$ corresponding to eigenvalues $\lambda_1 $ and $\lambda_2$ where $\lambda_1 \ne \lambda_2$.

Now,

$AX_1 = \lambda_1 X_1$ and $AX_2 = \lambda_2 X_2$.

Taking Transpose of first,

$ \begin{align} &(AX_1)^T = (\lambda_1 X_1)^T\\ \implies & X_1^TA^T = \lambda_1 X_1^T\\ \implies &X_1^TA^TX_2 = \lambda_1 X_1^T X_2\\ \implies & X_1^T\lambda_2 X_2 = \lambda_1 X_1^T X_2\\ \implies &\lambda_2X_1^T X_2 = \lambda_1 X_1^T X_2\\ \implies &(\lambda_2 - \lambda_1)X_1^T X_2 = 0\\ \end{align} $

Since $\lambda_2 \ne \lambda_1$, $X_1^T X_2 $ must be $0$. This establishes orthogonality of eigenvectors corresponding to two different eigenvalues.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.