1
$\begingroup$

Problem

I have a matrix $$X = diag(x_1, x_2, x_3), \quad x_i \sim N \left( 0, 1 \right) \quad i = \overline{1, 3}$$ and $x_i$ are independent.

SVD is used to find singular vectors: $$X = U \cdot \Sigma \cdot V^T,$$ where $V$ and $U$ are orthogonal matrices and $\Sigma$ is diagonal matrix, such that $$\Sigma = diag\left(\sigma_1, \sigma_2, \sigma_3\right), \, \sigma_1 \geq \sigma_2 \geq \sigma_3.$$

I need to find distribution of matrix $V \cdot U^T$. But first I need to find distribution of left and right singular vectors.

In question Singular vector of random Gaussian matrix on this forum that singular vectors have uniform distribution on the sphere of radius $1$. But I don't understand how to prove it.

My solution (probably wrong)

I tried to show that distribution of $X$ is invariant with respect to rotations. To do this we need an orthogonal matrix $A \, : \, A^T = A^{-1}$ and form new matrix $$X' = A \cdot X,$$ so that distribution of $X$ and $X'$ would be the same. Though, I've found out that they don't have the same distribution. $$x_{ij}' = \sum \limits_{s = 1}^s a_{is} \cdot x_{sj} = a_{ij} \cdot x_{jj}$$ as the matrix $X$ is diagonal. Then the mathematical expectation $$\mathbb{E} x_{ij}' = a_{ij} \cdot \mathbb{E} x_{jj} = a_{ij} \cdot \mathbb{E} x_j =0$$ as $\mathbb{E} x_i = 0 \, i = \overline{1, 3}$. But the variance of $x_{ij}'$ doesn't equal 1: $$Dx_{ij}' = a_{ij}^2 \cdot Dx_{jj} = a_{ij}^2 \cdot Dx_j = a_{ij}^2 \leq 1.$$ It can be equal to $1$, but not always.

Questions

What is the right way to prove the statement, that singular vectors have uniform distribution on the sphere of radius $1$?

Will the distribution of singular vectors change if diagonal matrix elements $x_i$ has normal distribution with expectation $a$ and variance $\sigma^2 \, : \, x_i \sim N \left( a, \sigma^2 \right) $? Am I right that then sphere would have radius $a$? What if matrix $X$ isn't diagonal?

$\endgroup$
2
  • $\begingroup$ The vector $x = (x_1, x_2, x_3)$ follows the multivariate normal distribution $N(\mu, \Sigma)$ with $\mu = 0$ and $\Sigma = I_3$, the $3 \times 3$ identity matrix. If $y = Ax$, then $y$ follows the multivariate normal distribution $N(A\mu, A \Sigma A^T)$ (does that look familiar)? The density of a multivariate normal variable $x \sim N(\mu, \Sigma)$ is proportional to $\exp\left[ -\frac{1}{2} (x - \mu)^T \Sigma^{-1} (x - \mu) \right]$. What is the density of $y$? That shows that $X$ is invariant under rotations $\endgroup$ Mar 16, 2018 at 21:00
  • $\begingroup$ @JonWarneke Thanks, it helped to understand diagonal matrix case. If the vector $x = \left( x_1, x_2, x_3 \right)$ follows the multivariate standard normal distribution $N\left( 0, I_3 \right)$ then density of $y = A \cdot x$ is the same as density of $x$. If normal distribution is not standard, then the density of $y$ is proportional to $exp \left\{ -\frac{1}{2} \left( y -A \mu \right)^T \left( A\Sigma A^T\right)^{-1} \left( y - A\mu \right) \right\}$. Distributions of $x$ and $y$ are not the same. What to do in this case? $\endgroup$ Mar 17, 2018 at 12:21

1 Answer 1

2
$\begingroup$

First, let's write down the probability density function (PDF) of the matrix $X$; this is just the joint density of $x_1, x_2, x_3$. Since those are independent normals, we have $$ \prod_{i=1}^3 \frac{1}{\sqrt{2\pi}} e^{-\frac{1}{2} x_i^2} \, dx_i = \frac{1}{(\sqrt{2\pi})^3} e^{-\frac{1}{2} (x_1^2+x_2^2 + x_3^2)} \, dx_1 dx_2 dx_3 = \frac{1}{(\sqrt{2\pi})^3} e^{-\frac{1}{2} \operatorname{tr}(X X^T)} \, dX. $$ Now, the singular value decomposition $X = U \Sigma V^T$ is actually a change of variables (up to ordering the singular values and signs of rows/columns of $U, V^T$, but let's gloss over that) $X \to (U, \Sigma, V^T)$. In other words, you can go from the probability density of the variable $X$ to the probability density of the three variables $U, \Sigma, V^T$. To do so, substitute $X = U \Sigma V^T$ into the PDF of $X$ to obtain $$ \frac{1}{(\sqrt{2\pi})^3} e^{-\frac{1}{2} \operatorname{tr}(X X^T)} \, dX = \frac{1}{(\sqrt{2\pi})^3} e^{-\frac{1}{2} \operatorname{tr}(\Sigma^T \Sigma)} \, dX $$ using the cyclic property of $\operatorname{tr}$ and the fact that $U, V$ belong to the orthogonal group.

Recall that the differentials $dX$ and $dU \, d\Sigma \, dV$ are related by the determinant of the Jacobian matrix for the change of variables; according to this, we have $$ dX = \prod_{1 \leq i < j \leq 3} (\sigma_i^2 - \sigma_j^2) \, d\Sigma \, dU \, dV, $$ so substituting gives $$ \frac{1}{(\sqrt{2\pi})^3} e^{-\frac{1}{2} \operatorname{tr}(X X^T)} \, dX = \frac{1}{(\sqrt{2\pi})^3} e^{-\frac{1}{2} \operatorname{tr}(\Sigma^T \Sigma)} \prod_{1 \leq i < j \leq 3} (\sigma_i^2 - \sigma_j^2) \, d\Sigma \, dU \, dV. $$ Now, note that the last term is the product of three PDFs (up to normalization): one for $\Sigma$ (namely $e^{-\frac{1}{2} \operatorname{tr}(\Sigma^T \Sigma)} \prod_{1 \leq i < j \leq 3} (\sigma_i^2 - \sigma_j^2) \, d\Sigma$), one for $U$ (namely $1 \, dU$), and one for $V$ (namely $1 \, dV$). Hence, those random variables (matrices) are independent, and moreover, $U$ and $V$ are distributed uniformly over the orthogonal group. Since $U$ and $V$ are independent and uniformly distributed over the orthogonal group, so is $V U^T$.

$\endgroup$
5
  • $\begingroup$ Thank you for the answer. Is it true to equate the density of three-dimensional vector to the density of a matrix? $\endgroup$ Mar 21, 2018 at 18:36
  • $\begingroup$ Yes — the density of a matrix is the joint density of its entries $\endgroup$ Mar 24, 2018 at 15:28
  • $\begingroup$ In article that you attached to the answer there are wedge products in $dX$. I have also found the same expression with wedge products for $dX$ in the book. But you have wrote $dX$ in more simple form and without wedge products. Can you explain, why it is possible? $\endgroup$ Apr 1, 2018 at 11:59
  • $\begingroup$ From the linked document, are you missing a factor of $U$ and $V'$ in front of the differentials in the Jacobian of the SVD? If so then do we no longer have that $U$ and $BV$ are uniformly distributed over the orthogonal group? $\endgroup$
    – gwtw14
    Mar 11, 2023 at 22:42
  • $\begingroup$ Update: I'm not too familiar with the use of forms in this context, but it seems that (V' dV) ^(wedge) (as in the notation in the linked doc) is the Haar measure, so the conclusion may still be correct $\endgroup$
    – gwtw14
    Mar 11, 2023 at 23:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .