Intuitive understanding of SVD for a matrix

Question

Could any give an intuitive understanding of SVD decomposition of a matrix? I know it can be used for image compress. But how to understand the decomposition within linear transform?

Answer 1

So let's examine the SVD $A=U\Sigma V^*$ where everything is chosen to be square $n\times n$ matrices (by padding $\Sigma$ with zeros and completing $V$ to be a unitary matrix.)

In this case, it is exactly what it looks like: $A$ is expressed as a rotation, followed by a scaling, followed by another rotation.

Another useful interpretation is that if you view the effects of $A$ on the unit sphere in $\Bbb R^n$, it will in general be changed into some ellipsoid (often a lower dimensional ellipsoid.) The singular values are the measurements of the minor axes of the ellipsoid. This in effect displays the extent the sphere was stretched in different directions. The 0 singular values correspond to dimensions that were "crushed flat" by the transformation, rather than shrunk/stretched.

Answer 2

Generally, an $N\times N$ matrix can be seen as a point in a high dimensional $N^2$ space. For instance, if any image would have only three pixels, we could've put any image on a point in a 3D space. SVD gives the primary "directions" that this matrix represents. The center matrix gives the weight of each direction in descending order, ie if you want to get a single direction that your matrix "goes in" (maybe think of some high dimensional gaussian blob that has a distinct direction), you can take only this direction. The direction itself is found in the left-hand matrix.

You can try to play with simple matrices such as $\begin{bmatrix}0&1\\1&0\end{bmatrix}$, $\begin{bmatrix}0&1\\1&1\end{bmatrix}$, $\begin{bmatrix}2&2\\1&-1\end{bmatrix}$ etc., and look at the center matrix and left hand matrix.