4
$\begingroup$

Suppose there is a given matrix $M$ of $m \times n$ dimensions. What is the relationship (if any) of its covariance matrix and its SVD's singular values matrix?

This page says that "the SVD represents an expansion of the original data in a coordinate system where the covariance matrix is diagonal", but I can't understand it clearly.

Thanks!

$\endgroup$

1 Answer 1

9
$\begingroup$

SVD exists for any matrix and is just one type of decomposition:

$$M=U\Sigma V^T$$ where $U$ is $m\times n$ orthogonal matrix, $\Sigma$ is $n \times n$ diagonal positive definite matrix (by convention the diagonal elements are in descending order) and $V$ is an orthogonal $n\times n$ matrix. Sometimes, $\Sigma$ is expanded to a rectangular form and $U$ is a complete basis of left space ($m\times m$).

Covariance matrix is something that only makes sense when rows of matrix $M$ represent basis functions evaluated at different datapoints. Essentially, columns are discrete functions (vectors) and covariance matrix summarizes how close to orthogonal they are (if they are orthogonal, the covariance matrix is diagonal -- entries outside the diagonal measure the correlation between columns).

Let's put SVD decomposition into the covariance matrix:

$$C=M^T M=V\Sigma^2 V^T$$

$\Sigma^2$ is obviously diagonal, so SVD can help you compute the eigenvalue decomposition of the covariance matrix. The statement you don't understand is just a written form of above equation. But let's observe what this means in terms of columns of $M$. If the columns of $M$ were vectors: $M_{ij}=\vec{v}^{j}_i$, then $C_{ij}=\vec{v}^{(i)}\cdot\vec{v}^{(j)}$. $V$ tells you which linear combinations of columns you must take in order to make the resulting rows orthogonal to each other (nondiagonal terms become $0$), and does so by using $V$ that is also orthogonal by itself. These linear combinations of rows are retrieved by doing $M'=MV$. It follows $M'=U\Sigma$ and therefore $M'^T M=\Sigma U^T U\Sigma = \Sigma^2$. If $M$ is a matrix for fitting and columns refer to trial functions, $V$ tells you which linear combinations of these functions will make the covariance matrix diagonal.

For instance, if your trial functions are $1$ and $x$, and $V=\frac{1}{\sqrt{2}}\begin{bmatrix}1 & 1\\ 1& -1\end{bmatrix}$, then taking the basis $(1+x)/\sqrt2 $ and $(1-x)/\sqrt2$ will result in a diagonal covariance matrix.

$\endgroup$
2
  • $\begingroup$ And what about a relationship between a given covariance matrix $\mathbf{C}$ and it's SVD? And also how does an SVD of a covariance matrix related to the Cholesky decomposition of that covariance matrix $\mathbf{C} = \mathbf{LL^T}$? I think that $\mathbf{L}$ can be used to induce a covariance structure of some uncorrelated data $\mathbf{X}$ as $\mathbf{Y} = \mathbf{LX}$. Can SVD matrices be used to do the same? What is the relationship between all of these? Thank you. $\endgroup$
    – Confounded
    Commented Nov 18, 2021 at 10:36
  • $\begingroup$ PS. Is it the case that if $\mathbf{C}$ is a covariance matirx, then it's SVD is given by only two matrices, not three, and that it is given by $\mathbf{C} = \mathbf{U \Sigma U}^T$ with a diagonal $\mathbf{\Sigma}$ and orthogonal $\mathbf{U}$? If the diagonal $\mathbf{\Sigma}$ simply has the variances on the diagonal? And is the Cholesky matrix is then given by $\mathbf{L} = \mathbf{U \sqrt{\Sigma} V}^T$ with some other orthogonal matrix $\mathbf{V}$? $\endgroup$
    – Confounded
    Commented Nov 18, 2021 at 11:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .