Suppose $\Omega$ is a Gaussian matrix with entries distributed i.i.d. according to normal distribution $\mathcal{N}(0,1)$. Let $U \Sigma V^{\mathsf T}$ be its singular value decomposition. What would be the distribution of the column (or row) vectors of $U$ and $V$? Would it be a Gaussian or anything closely related?

  • $\begingroup$ $U$ and $V$ are matrices with orthonormal columns, and since in this case they have as many columns as rows, they are orthogonal matrices. Since the norm of each column must be $1$, the entries in the column cannot be independent and cannot be normally distributed. The entries in a random vector uniformly distributed on the sphere in $\mathbb R^n$ are approximately normal if $n$ is large, and not if $n$ is small. $\endgroup$ – Michael Hardy Jul 18 '14 at 14:29

Suppose the entries in a random matrix $\Omega\in\mathbb R^{n\times n}$ are i.i.d. $N(0,1)$. Then for any orthogonal matrices $G,H\in\mathbb R^{n\times n}$, the entries in $G\Omega H^T$ are also i.i.d. normal. Maybe the easiest way to see that is is this: let $\operatorname{vec}(M)$ be an $n^2\times 1$ vector in which the entries in $M\in\mathbb R^{n\times n}$ are stacked up into one column: the first column of $M$ followed by the second, and so on. Then $\operatorname{vec}(G\Omega H^T) = (G\otimes H)\operatorname{vec}(\Omega)$, where $G\otimes H$ is a Kronecker product, and then observe that $G\otimes H$ is also an orthogonal matrix. When you multiply a random vector with i.i.d. $N(0,1)$ entries by an orthogonal matrix, you get another random vector with i.i.d. $N(0,1)$ entries. That is because $\operatorname{var}(A\Omega) = A\Big(\operatorname{var}(\Omega)\Big) A^T$, so you have the variance $AIA^T = I$.

So this gives us an argument from spherical symmetry: the distribution of the singular vectors of $\Omega$ must be the same as the distribution of the singular vectors of $G\Omega H^T$; hence the distribution is invariant under rotations. Consequently the distibution of each singular vector is uniform on the sphere of radius $1$ centered at $0$ in $\mathbb R^n$.

The distribution of the singular values is a harder problem. The distribution of the singular values is of interest in statistics: Suppose you suspect that the matrix you're looking at is a low-rank matrix plus random noise, so that only the noise accounts for the data matrix having full rank. Then you might expect some large singular values that would have been nonzero even if there had been no noise, plus some small singular values that are nonzero only because of the noise. So the question is: when do the singular values of the data matrix differ significantly from those coming from your matrix $\Omega$?

  • 1
    $\begingroup$ Sorry to reanimate an old topic, but one of your claims seems unclear to me: "Consequently the distibution of each singular vector is uniform on the sphere of radius 1 centered at 0 in $\mathbb{R}^n$ ". Due to singular vector orthogonality, you must be talking about marginal distributions of each vector; can we say something about their joint distribution (i.e. distribution of matrix $U$)? $\endgroup$ – Vossler Jan 17 '16 at 17:09

The original question suggested IID entries, which means that in the limit that the matrix gets big, the singular values follow a Marchenko-Pastur distribution.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.