# What does singular value decomposition of covariance matrix represent?

I am reading the paper "Understanding dimensional collapse in contrastive self-supervised learning." The authors identified a dimensional collapse phenomenon: i.e. some dimension of embedding collapses to zero. They show this by collecting the embedding vectors on the validation set. Each embedding vector has a size of $$d=128$$, then compute the covariance matrix $$C\in\mathbb{R}^{d\times d}$$. Then the singular value decomposition is applied on the covariance matrix. They state that a number of singular values collapse to zero, thus representing collapsed dimensions.

Thus my questions are:

1. What does singular value decomposition of covariance matrix represent?
2. Why a number of singular value of covariance matrix collapse to zero can represent these dimension of embedding collapse?
• Does this answer your question? What is the intuitive relationship between SVD and PCA? Apr 20, 2022 at 17:51
• Nope, The paper use the SVD of covariance.
– Noel
Apr 22, 2022 at 0:52
• "The paper use the SVD of covariance". The title of your question is "What does the singular value decomposition of covariance matrix represent?" . I was under the impression that SVD is the abbreviation of singular value decomposition. Please tell me now why exactly your question is not a duplicate? Apr 22, 2022 at 4:19
• The answer said that the eigenvalue decomposition on covariance matrix is equivalent to SVD or original matrix. But I am confused by applying SVD on covariance matrix rather than original matrix. Thanks again.
– Noel
Apr 22, 2022 at 8:10
• Sure SVD is normally applied to the data matrix. Regarding your question: Isn't a PCA (diagonalization of the covariance matrix) not just a special case of an SVD of it ? It is probably time to post a link to the paper you are reading. The images tell me nothing. Apr 22, 2022 at 8:39

Keep in mind that because all covariance matrix are symmetric and positive semi-definite, their singular values are the same as their eigenvalues. So you don't actually need to compute the SVD and can just directly compute the eigenvalues if you are interested in a rotation invariant measure of scale. As for $$U$$ and $$V$$, You can think of $$V$$ as rotating such that the axes are aligned to the major axes of variance, then we apply the eigenvalues / singular values to scale them. Because the covariance matrix is symmetric and positive semi-definite, $$U = V$$. I don't have a proof for it, but at least in this case, $$U$$ and $$V$$ appear to be involutory (their own inverse). So $$U$$ rotates back to the original space which isn't aligned on the major axes of variance.