orthogonal eigenvectors

I have a very simple question that can be stated without proof. Are all eigenvectors, of any matrix, always orthogonal? I am trying to understand Principal components and it is cruucial for me to see the basis of eigenvectors.

-
No. Take any any non-orthogonal basis $(v_1,\dots,v_n)$ and define a linear map $A$ on this basis by sending each $v_i$ to $iv_i$. The eigenspaces are the $n$ lines generated by the $v_i$, and these are by construction not ortgogonal. –  Olivier Bégassat May 8 '12 at 13:38
Eigenvectors corresponding to different eigenvalues will be orthogonal if the matrix is symmetric. This is part of the real spectral theorem. –  Dylan Moreland May 8 '12 at 13:38
The case @Dylan is describing will apply to your study of principal components, since the underlying matrices are symmetric... –  Ｊ. Ｍ. May 8 '12 at 14:50

Fix two linearly independent vectors $u$ and $v$ in $\mathbb{R}^2$, define $Tu=u$ and $Tv=2v$. Then extend linearly $T$ to a map from $\mathbb{R}^n$ to itself. The eigenvectors of $T$ are $u$ and $v$ (or any multiple). Of course, $u$ need not be perpendicular to $v$.

-
Thank you very much for your answers. Could anyone state whether they are orthogonal in PCA case? –  Bober02 May 8 '12 at 13:45
For PCA, things can always be set up such that the eigenvectors are orthogonal. On the other hand, I would recommend looking at PCA as a singular value decomposition instead of as an eigendecomposition. It's been discussed here on math.SE a number of times; search around. –  Ｊ. Ｍ. May 8 '12 at 14:48
What do you mean by "setting up"? Is there some common technique to achive a singular matrix? –  Bober02 May 9 '12 at 8:12
My understanding based on en.wikipedia.org/wiki/Principal_component_analysis is that a singular value decomposition (SVD) of $X$ results in three matrices, where the first of them is composed of the eigenvectors of $X^T X$, and is used for PCA. This matrix is always orthogonal. –  Uri Jun 14 at 13:04
Beautiful answer. –  Don Larynx Nov 13 at 15:14