Understanding singular value decomposition

Question

Please, would someone be so kind and explain what exactly happens when Singular Value Decomposition is applied on a matrix? What are singular values, left singular, and right singular vectors? I know they are matrices of specific form, I know how to calculate it but I cannot understand their meaning.

I have recently been sort of catching up with Linear Algebra and matrix operations. I came across some techniques of matrix decomposition, particularly Singular Value Decomposition and I must admit I am having problem to understand the meaning of SVD.

I read a bit about eigenvalues and eigenvectors only because I was interested in PCA and I came across diagonalizing a covariance matrix which determines its eigenvectors and eigenvalues (to be variances) towards those eigenvectors. I finally understood it but SVD gives me really hard time.

thanks

Answer 1

One geometric interpretation of the singular values of a matrix is the following. Suppose $A$ is an $m\times n$ matrix (real valued, for simplicity). Think of it as a linear transformation $\mathbb R^n \to \mathbb R^m$ in the usual way. Now take the unit sphere $S$ in $\mathbb R^n$. Being a linear transformation, $A$ maps $S$ to an ellipsoid in $\mathbb R^m$. The lengths of the semi-axes of this ellipsoid are precisely the non-zero singular values of $A$. The zero singular values tell us what the dimension of the ellipsoid is going to be: $n$ minus the number of zero singular values.

Answer 2

Maybe it helps to think in terms of linear transformations rather than matrices. Suppose $V$ and $W$ are finite dimensional inner product spaces over $F$ (where $F$ is $\mathbb R$ or $\mathbb C$) and suppose that $T:V \to W$ is a linear transformation. Then, according to the SVD theorem, there exist orthonormal bases $\alpha$ and $\beta$ (bases of $V$ and $W$, respectively) such that $[T]_{\alpha}^{\beta}$ is diagonal.

Trefethen explains a nice geometrical interpretation of the SVD in his book Numerical Linear Algebra.

Answer 3

There is another geometric interpretation of the singular values of a matrix is the following. Suppose $A$ is an $m\times n$ matrix (real valued, for simplicity). Think of it as a linear transformation $\mathbb R^n \to \mathbb R^m$ in the usual way.

We know from Gauss elimination (see e.g. Gilbert Strang's book) that the range of the linear mapping $x \to A*x$, which is just the column space. has the same dimension as the row-space of $A$, or better the column-space of $A'$, the transpose (conjugate) matrix, which in turn is the orthogonal complemnet of the null space of $A$ (because each equation in the homogeneous linear system $A \ast x = 0$ refers to orthogonality to one of the row vectors, so the whole system is the same as orthogonality to all the row vectors, which is the same as orthogonality to their linear span (the row space).

Usually this is known by the formula: $rowrank(A) = colrank(A)=: rank(A)$.

So the mapping $x \to A \ast x$, when restricted to the column-space of $A'$ does not have a null space, and thus establishes an isomorphism between the row-space and the column space of $A$.

So essentially a matrix $A$ of rank $r$ is in the most general form an isomorphism between to specific $r$-dimensonal subspaces of $\mathbb R^n$ and $\mathbb R^m$ respectively.

The remarkable fact then expressed by the SVD theorem is that one can find two orthonormal bases for these two $r$-dimensional spaces such that the linear mapping $x \to A \ast x$ can be described by a diagonal matrix with non-negative diagonal entries on these spaces. This comes from the spectral theorem applied to the semi-positive matrices $A' \ast A$ or $A \ast A'$.

Answer 4

Think of a matrix $A$ as a transformation of space. SVD is a way to decompose this transformation into a series of three consecutive, canonical transformations: a first rotation, scaling and a second rotation. There is a nice picture on Wikipedia showing this transformation:

Another thing that helps to build the intuition is to derive the equation yourself. For any matrix $A$ of size $n\times m$ the row space and a column space will be subspaces of $R^m$ and $R^n$ respectively. You are looking now for a orthonormal basis of a row space of $A$, say $V$, that gets transformed into the column space of $A$ such that it remains an orthonormal basis (in the column space of $A$). Let's call this transformed basis $U$. So each $v_i$ get's mapped to some $u_i$, possibly with some stretching, say scaled by $\sigma_i$: $$ Av_i = u_i \sigma_i $$ In matrix notation we get: $$ AV = U \Sigma $$ Now, we can multiply (from the right) both sides by $V^{-1}$, and knowing that $V^{-1} = V^{T}$ (since for an orthonormal basis $V^TV = I$) we get: $$ A = U \Sigma V^T $$

Side notes:

• SVD works both for real and complex matrices, so in general $A = U \Sigma V^*$, where $V^*$ is a conjugate transpose of $V$.

• SVD is a generalisation of a Spectral Decomposition, which is also a diagonal factorisation, but for symmetric matrices only (or, more specifically, Hermitian).

Answer 5

Answer referring to Linear Algebra from the book Deep Learning by Ian Goodfellow and 2 others.

• The Singular Value Decomposition (SVD) provides a way to factorize a matrix, into singular vectors and singular values. Similar to the way that we factorize an integer into its prime factors to learn about the integer, we decompose any matrix into corresponding singular vectors and singular values to understand behaviour of that matrix.

• SVD can be applied even if the matrix is not square, unlike Eigendecomposition (another form of decomposing a matrix).

• SVD of any matrix A is given by: A = UDV.T (transpose of V) The matrix U and V are orthogonal matrices, D is a diagonal matrix (not necessarily square).

• Elements along diagonal D are known as Singular values. The columns of U are known as the left-singular vectors. The columns of V are known as right-singular vectors.

• The most useful feature of the SVD is that we can use it to partially generalize matrix inversion to non-square matrices