85
$\begingroup$

I have a very simple question that can be stated without any proof. Are all eigenvectors, of any matrix, always orthogonal? I am trying to understand principal components and it is crucial for me to see the basis of eigenvectors.

$\endgroup$
5
  • 11
    $\begingroup$ No. Take any any non-orthogonal basis $(v_1,\dots,v_n)$ and define a linear map $A$ on this basis by sending each $v_i$ to $iv_i$. The eigenspaces are the $n$ lines generated by the $v_i$, and these are by construction not ortgogonal. $\endgroup$ Commented May 8, 2012 at 13:38
  • 16
    $\begingroup$ Eigenvectors corresponding to different eigenvalues will be orthogonal if the matrix is symmetric. This is part of the real spectral theorem. $\endgroup$ Commented May 8, 2012 at 13:38
  • 2
    $\begingroup$ The case @Dylan is describing will apply to your study of principal components, since the underlying matrices are symmetric... $\endgroup$ Commented May 8, 2012 at 14:50
  • $\begingroup$ @PL this is clearly not a duplicate of that question - this question is about the general case when $M$ is not necessarily a real symmetric matrix. $\endgroup$
    – KReiser
    Commented Jul 26, 2020 at 1:55
  • $\begingroup$ @Dylan Moreland I have a contradiction to this $\endgroup$
    – D. Sikilai
    Commented Apr 11 at 3:40

5 Answers 5

78
$\begingroup$

In general, for any matrix, the eigenvectors are NOT always orthogonal. But for a special type of matrix, symmetric matrix, the eigenvalues are always real and eigenvectors corresponding to distinct eigenvalues are always orthogonal. If the eigenvalues are not distinct, an orthogonal basis for this eigenspace can be chosen using Gram-Schmidt.

For any matrix $M$ with $n$ rows and $m$ columns, $M$ multiplies with its transpose, either $M M'$ or $M'M$, results in a symmetric matrix, so for this symmetric matrix, the eigenvectors are always orthogonal.

In the application of PCA, a dataset of $n$ samples with $m$ features is usually represented in a $n\times m$ matrix $D$. The variance and covariance among those $m$ features can be represented by a $m\times m$ matrix $D'D$, which is symmetric (numbers on the diagonal represent the variance of each single feature, and the number on row $i$ column $j$ represents the covariance between feature $i$ and $j$). The PCA is applied on this symmetric matrix, so the eigenvectors are guaranteed to be orthogonal.

$\endgroup$
1
  • 1
    $\begingroup$ The qualification "symmetric" for a matrix should almost always be accompanied by "real", in cases where the notion is useful; that is the case for this answer. Also, one does not need the Gram-Schmidt procedure to choose an orthogonal basis; it is only useful to correct the inadvertent choice of a non-orthogonal basis. If in the end you need an orthogonal basis, just add the condition of orthogonality to your system every time you choose a basis vector before choosing another one, and you basis will be orthogonal straight away. $\endgroup$ Commented Jul 30, 2023 at 9:34
34
$\begingroup$

Fix two linearly independent vectors $u$ and $v$ in $\mathbb{R}^2$, define $Tu=u$ and $Tv=2v$. Then extend linearly $T$ to a map from $\mathbb{R}^n$ to itself. The eigenvectors of $T$ are $u$ and $v$ (or any multiple). Of course, $u$ need not be perpendicular to $v$.

$\endgroup$
6
  • $\begingroup$ Thank you very much for your answers. Could anyone state whether they are orthogonal in PCA case? $\endgroup$
    – Bober02
    Commented May 8, 2012 at 13:45
  • 2
    $\begingroup$ For PCA, things can always be set up such that the eigenvectors are orthogonal. On the other hand, I would recommend looking at PCA as a singular value decomposition instead of as an eigendecomposition. It's been discussed here on math.SE a number of times; search around. $\endgroup$ Commented May 8, 2012 at 14:48
  • $\begingroup$ What do you mean by "setting up"? Is there some common technique to achive a singular matrix? $\endgroup$
    – Bober02
    Commented May 9, 2012 at 8:12
  • 3
    $\begingroup$ My understanding based on en.wikipedia.org/wiki/Principal_component_analysis is that a singular value decomposition (SVD) of $X$ results in three matrices, where the first of them is composed of the eigenvectors of $X^T X$, and is used for PCA. This matrix is always orthogonal. $\endgroup$
    – Uri
    Commented Jun 14, 2013 at 13:04
  • $\begingroup$ Beautiful answer. $\endgroup$
    – Don Larynx
    Commented Nov 13, 2013 at 15:14
9
$\begingroup$

Not necessarily all orthogonal. However two eigenvectors corresponding to different eigenvalues are orthogonal, whenever the matrix is symmetric.

e.g Let $X_1$ and $X_2$ be two eigenvectors of a matrix $A$ corresponding to eigenvalues $\lambda_1 $ and $\lambda_2$ where $\lambda_1 \ne \lambda_2$.

Now,

$AX_1 = \lambda_1 X_1$ and $AX_2 = \lambda_2 X_2$.

Taking Transpose of first,

$ \begin{align} &(AX_1)^T = (\lambda_1 X_1)^T\\ \implies & X_1^TA^T = \lambda_1 X_1^T\\ \implies &X_1^TA^TX_2 = \lambda_1 X_1^T X_2\\ \implies & X_1^T\lambda_2 X_2 = \lambda_1 X_1^T X_2\\ \implies &\lambda_2X_1^T X_2 = \lambda_1 X_1^T X_2\\ \implies &(\lambda_2 - \lambda_1)X_1^T X_2 = 0\\ \end{align} $

Since $\lambda_2 \ne \lambda_1$, $X_1^T X_2 $ must be $0$. This establishes orthogonality of eigenvectors corresponding to two different eigenvalues.

$\endgroup$
3
  • 11
    $\begingroup$ It's implicitly assumed that A is symmetric. $\endgroup$
    – FuzzyDuck
    Commented Dec 24, 2021 at 18:32
  • $\begingroup$ Does this hold when A is not symmetric? $\endgroup$
    – Andy R
    Commented Jul 3, 2023 at 12:55
  • 2
    $\begingroup$ @AndyR it holds to a larger class of matrices called normal matrices $\endgroup$ Commented Jul 30, 2023 at 8:31
5
$\begingroup$

In the context of PCA: it is usually applied to a positive semi-definite matrix, such as a matrix cross product, $X ' X$, or a covariance or correlation matrix.

In this PSD case, all eigenvalues, $\lambda_i \ge 0$ and if $\lambda_i \ne \lambda_j$, then the corresponding eivenvectors are orthogonal. If $\lambda_i = \lambda_j$ then any two orthogonal vectors serve as eigenvectors for that subspace.

$\endgroup$
1
$\begingroup$

Not only are the eigenvectors not generally orthogonal, they're not always uniquely defined. For example, any basis whatever for the space constitutes a set of eigenvectors for the eigenvalue 1 of the identity matrix.

You can orthogonalize the eigenvectors to taste --- but orthogonality plays no role in their being eigenvectors.

As the other posters pointed out, there are special classes of matrices where orthogonality is involved in the eigenanalysis.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .