Suppose there is a given matrix $M$ of $m \times n$ dimensions. What is the relationship (if any) of its covariance matrix and its SVD's singular values matrix?
This page says that "the SVD represents an expansion of the original data in a coordinate system where the covariance matrix is diagonal", but I can't understand it clearly.
Thanks!
SVD exists for any matrix and is just one type of decomposition:
$$M=U\Sigma V^T$$ where $U$ is $m\times n$ orthogonal matrix, $\Sigma$ is $n \times n$ diagonal positive definite matrix (by convention the diagonal elements are in descending order) and $V$ is an orthogonal $n\times n$ matrix. Sometimes, $\Sigma$ is expanded to a rectangular form and $U$ is a complete basis of left space ($m\times m$).
Covariance matrix is something that only makes sense when rows of matrix $M$ represent basis functions evaluated at different datapoints. Essentially, columns are discrete functions (vectors) and covariance matrix summarizes how close to orthogonal they are (if they are orthogonal, the covariance matrix is diagonal -- entries outside the diagonal measure the correlation between columns).
Let's put SVD decomposition into the covariance matrix:
$$C=M^T M=V\Sigma^2 V^T$$
$\Sigma^2$ is obviously diagonal, so SVD can help you compute the eigenvalue decomposition of the covariance matrix. The statement you don't understand is just a written form of above equation. But let's observe what this means in terms of columns of $M$. If the columns of $M$ were vectors: $M_{ij}=\vec{v}^{j}_i$, then $C_{ij}=\vec{v}^{(i)}\cdot\vec{v}^{(j)}$. $V$ tells you which linear combinations of columns you must take in order to make the resulting rows orthogonal to each other (nondiagonal terms become $0$), and does so by using $V$ that is also orthogonal by itself. These linear combinations of rows are retrieved by doing $M'=MV$. It follows $M'=U\Sigma$ and therefore $M'^T M=\Sigma U^T U\Sigma = \Sigma^2$. If $M$ is a matrix for fitting and columns refer to trial functions, $V$ tells you which linear combinations of these functions will make the covariance matrix diagonal.
For instance, if your trial functions are $1$ and $x$, and $V=\frac{1}{\sqrt{2}}\begin{bmatrix}1 & 1\\ 1& -1\end{bmatrix}$, then taking the basis $(1+x)/\sqrt2 $ and $(1-x)/\sqrt2$ will result in a diagonal covariance matrix.
