# Intuitively, what is the difference between Eigendecomposition and Singular Value Decomposition?

I'm trying to intuitively understand the difference between SVD and eigendecomposition.

From my understanding, eigendecomposition seeks to describe a linear transformation as a sequence of three basic operations ($P^{-1}DP$) on a vector:

1. Rotation of the coordinate system (change of basis): $P$
2. Independent scaling along each basis vector (of the rotated system): $D$
3. De-rotation of the coordinate system (undo change of basis): $P^{-1}$

But as far as I can see, SVD's goal is to do exactly the same thing, except that resulting decomposition is somehow different.

What, then, is the conceptual difference between the two?

For example:

• Is one of them more general than the other?
• Is either a special case of the other?

Note: I'm specifically looking for an intuitive explanation, not a mathematical one.
Wikipedia is already excellent at explaining the mathematical relationship between the two decompositions ("The right-singular vectors of M are eigenvectors of $M^*M$", for example), but it completely fails to give me any intuitive understanding of what is going on intuitively.

The best explanation I've found so far is this one, which is great, except it doesn't talk about eigendecompositions at all, which leaves me confused as to how SVD is any different from eigendecomposition in its goal.

-

Intuitively, $SVD$ says for any linear map, there is an orthonormal frame in the domain such that it is first mapped to a different orthonormal frame in the image space, and then the values are scaled.

Eigendecomposition says that there is a basis, it doesn't have to be orthonormal, such that when the matrix is applied, this basis is simply scaled. That is assuming you have $n$ linearly independent eigenvectors of course. In some cases your eigenspaces may have the linear map behave more like upper triangular matrices.

Edit: Consider the difference for a rotation matrix in $\mathbb{R}^2$.

Here, there are no real eigenvalues and this corresponds to there being no choice of basis which under the transformation is simply a scaling. On the other hand, SVD makes a lot of sense here because it says we can take the standard basis in the domain, map it to the rotated version of this basis (thought of as in the image space), and scale everything by 1.

-
+1 Hmm, SVD preserves normality too? I thought it just preserves orthogonality, like when transforming this image into this image from the article I linked to. – Mehrdad Mar 4 '13 at 7:07
Also, regarding the rotation matrix example -- I think there is a basis in which the transformation is just a scaling, but the scaling is by a complex factor, right? – Mehrdad Mar 4 '13 at 7:11
It preserves normality until the point where you introduce the scaling. The basic idea is you take one orthonormal frame, map those to a different orthonormal frame, and then scale that second orthonormal frame (losing the normality). Extend everything else by linearity. As for the rotation matrix, yes there are complex eigenvalues corresponding to a complex eigenvector but I'm restricting the geometric interpretation to just vectors in $\mathbb{R}^2$ where these don't exist. – muzzlator Mar 4 '13 at 8:33
Let me put $SVD$ another way. Suppose for now $M$ is a square matrix. It says there exists two orthonormal bases $\{b_1, ..., b_n\}$ and $\{c_1, ..., c_n\}$ such that $M b_i = k_i c_i$. When $M$ is not square, it does something pretty similar, just forget about the vectors which are sent to $0$ or which aren't in the image. – muzzlator Mar 4 '13 at 8:43

Consider the eigendecomposition $A=P D P^{-1}$ and SVD $A=U \Sigma V^*$. Some key differences are as follows,

• The vectors in the eigendecomposition matrix $P$ are not necessarily orthogonal, so the change of basis isn't a simple rotation. On the other hand, the vectors in the matrices $U$ and $V$ in the SVD are orthonormal, so they do represent rotations (and possibly flips).
• In the SVD, the nondiagonal matrices $U$ and $V$ are not necessairily the inverse of one another. They are usually not related to each other at all. In the eigendecomposition the nondiagonal matrices $P$ and $P^{-1}$ are inverses of each other.
• In the SVD the entries in the diagonal matrix $\Sigma$ are all real and nonnegative. In the eigendecomposition, the entries of $D$ can be any complex number - negative, positive, imaginary, whatever.
• The SVD always exists for any sort of rectangular or square matrix, whereas the eigendecomposition can only exists for square matrices, and even among square matrices sometimes it doesn't exist.
-
+1 Hmm, so if the change of basis performed by $P$ isn't a simple rotation, then what else can it be? The scaling is already taken care of by $D$, right? I think an example might be helpful for that one. – Mehrdad Mar 4 '13 at 7:10
If $P=\begin{bmatrix}1 & 1 \\ 0 & 1\end{bmatrix}$, then the action of $P$ on a vector is to form a linear combination of the vectors $\begin{bmatrix}1 \\ 0\end{bmatrix}$ and $\begin{bmatrix}1 \\ 1\end{bmatrix}$, which make a 45 degree angle. The action of $P^{-1}$ is to express the vector acted on in the basis of those vectors. – Nick Alger Mar 4 '13 at 7:27

Edit: Thank you, John, for improving my newbie formatting. I hope there is a way to bold the vectors in math mode; I haven't found a way to do it yet.

I think I can provide a much clearer (and shorter) explanation of singular vectors versus eigenvectors.

First, I encourage you to see an $(m \times n)$ real-valued matrix $A$ as a bilinear operator between two spaces; intuitively, one space lies to the left ($R^m$) and the other ($R^n$) to the right of $A$. "Bilinear" simply means that $A$ is linear in both directions (left to right or right to left). The operations $A$ can perform are limited to scaling, rotation, and reflection, and combinations of these; any other kind of operation is non-linear.

$A$ transforms vectors between the two spaces via multiplication:

$x^T$ A = $y^T$ transforms left vector $x$ to right vector $y$.

$x = A y$ transforms right vector $y$ to left vector $x$.

The point of decompositions of $A$ is to identify, or highlight, aspects of the action of $A$ as an operator. The eigendecomposition of $A$ clarifies what $A$ does by finding the eigenvalues and eigenvectors that satisfy the constraint

$A x = \lambda x$.

This constraint identifies vectors (directions) that $A$ does not rotate, and the scalars $\lambda$ associated with each of those directions.

The problem with eigendecomposition is that when the left and right space are different spaces, there really isn't a sense in which $A$'s action can be described as involving a "rotation", because the left and right spaces are totally separate, not "oriented" relative to one another. There just isn't a way to generalize the notion of an eigendecomposition to a non-square matrix $A$.

Singular vectors provide a different way to identify vectors for which the action of $A$ is simple; one that does generalize to the case where the left and right spaces are different. A corresponding pair of singular vectors have a scalar $\sigma$ for which $A$ scales by the same amount, whether transforming from the left space to the right space or vice-versa:

$x^T A = \sigma y^T$

$\sigma x = A y$.

Thus, eigendecomposition represents $A$ in terms of how it scales vectors it doesn't rotate, while singular value decomposition represents $A$ in terms of corresponding vectors that are scaled the same, whether moving from the left to the right space or vice-versa. When the left and right space are the same (i.e. when $A$ is square), singular value decomposition represents $A$ in terms of how it rotates and reflects vectors that $A$ and $A^T$ scale by the same amount.

My original "answer" is below, however I hope you will agree that the one above is far superior.

Maybe it would help to have an intuitive (by which I mean a geometrical) explanation of what singular vectors and eigenvectors are.

A singular value $s$ of a square ($n \times n$) matrix $A$ is a nonzero scalar for which there exist unit vectors $x$ and $y$ that satisfy the two equations:

$A^T x = \sigma y$ and

$A y = \sigma x$.

$x$ and $y$ are the left and right singular vectors, respectively, of $A$. A geometric interpretation of this is as follows. Let $u(t) = t x$ and $v(t) = t y$ be the parametric equations defining the lines in the direction of the vectors $x$ and $y$, respectively. Then

$A^T u(t) = A^T tx = t A^T x = t \sigma y = v(\sigma t)$,

$A v(t) = A ty = t A y = t \sigma x = u(\sigma t)$.

In words, the bilinear operator $A$ does the following: $A^T$ maps the line $u(t)$ to the line $v(\sigma t)$, where the change in the parameter ($t$ to $\sigma t$) means "your speed along the line $v$ is multiplied by $\sigma$". Likewise, $A$ maps the line $v(t)$ to the line $u(\sigma t)$.

The eigenvectors of $A$ have a different interpretation: given any vector $x$,

(1) project $x$ onto the eigenvectors of $A$ (i.e. represent $x$ in the set of basis vectors that are the eigenvectors of $A$).

(2) scale each projected component of $x$ by the corresponding eigenvector; the resulting vector $y = A x$ (but $y$ is represented in the eigenvector basis).

(3) "un-project" the vector $y$ back to the original coordinate system.

Thus, the eigenvectors define a new set of basis vectors along which the scaling occurs; the singular vectors define lines that $A$ maps, one to the other.

I hope that's the kind of intuitive answer you were looking for. It certainly was the kind I was looking for.

-

I found a clear explanation for the SVD here:

The SVD allows to describe the effect of a matrix on a vector (via the matrix-vector product), as a three-step process:

1. A first rotation in the input space
2. A simple positive scaling that takes a vector in the input space to the output space
3. And another rotation in the output space
-

I see one problem with the question. Linear operator $A$ is is decomposable into $P^{-1}DP$, where $D$ is a diagonal matrix if and only if $A$ is full rank. Otherwise $D$ has to be non-diagonal matrix in Jordan normal form.

The 'advantage' of SVD is that it always 'diagonalizes' a matrix even if it is non-square.

-