2
$\begingroup$

Say I have some M-dimensional joint probability distribution F (let's say it's a multivariate gaussian) and I repeat the following steps multiple times:

  • draw N samples (M-element vectors) from F
  • stack these vectors to form the rows of the NxM matrix A
  • compute the right singular vectors of A

Each time I perform these steps, A is a different matrix and I will of course get different right singular vectors. However, if N >> M, and the variances in F are relatively small, it makes intuitive sense that the set of right singular vectors will be "similar" between multiple runs of this processes. Moreover, if I repeat this process with a larger value of N, it seems likely that the average degree of "similarity" that I would observe between runs would be larger.

However, I'm not sure how to formalize this intuition. Perhaps as I increase the number of rows in A, the distribution of the right singular vectors will converge in probability to some known probability distribution? Perhaps there some meaningful metric of similarity that decreases as the number of rows increases?

I've encountered this question Singular vector of random Gaussian matrix and this question How to prove that singular vectors have uniform distribution on the sphere?, which seem to suggest that in the case that the multivariate gaussian is zero mean with an identity covariance matrix the right singular vectors are uniformly distributed, but I'm not sure:

  • whether this still applies in the case of non-zero mean and non-unit/-diagonal covariance matrix
  • how the characteristics of this uniform distribution change as I increase the number of rows of A

Any help, or even pointers in the right direction would be much appreciated! Thank you!

$\endgroup$
0
$\begingroup$

Not an answer. Just some thoughts for large $N$ case.

Let the column vector $x^{(i)} \in \mathbb{R}^{M\times 1}$ be the $i$th sample $(1 \le i \le N)$, and $x^{(i)}_j \in \mathbb{R}$ be its $j$th entry $(1 \le j \le M)$. Then by construction of $A_{N\times M}$, we have: $A_{ij} = x^{(i)}_j$.

The right singular vectors of $A$ are the (right) eigenvectors of $C = A^T A$, which is an $M \times M$ matrix. We will consider a rescaled version $B = {1\over N} C$. Clearly this rescaling affects the eigenvalues but not the eigenvectors (for any $N$, not just in the large $N$ limit). Now:

$$B_{kl} = {1\over N} \sum^N_{p=1} (A^T)_{kp} A_{pl} = {1\over N}\sum^N_{p=1} A_{pk} A_{pl} = {1\over N}\sum^N_{p=1} x^{(p)}_k x^{(p)}_l=\overline{x_k x_l}$$

where the overline symbol $\overline{x_k x_l}$ denotes the sample mean of $x_k x_l$, i.e. averaged over the $N$ samples. For large $N, \overline{x_k x_l} \rightarrow \mathbb{E}[x_k x_l]$, and so:

$$ B \rightarrow \mathbb{E}[x\ x^T] $$

At this point my argument becomes very informal - sorry in advance!

  • In the special case of each $x^{(i)}_j \sim N(0,1)$ independently, then $\mathbb{E}[x\ x^T] = I_{M \times M}$ and you cannot say anything about the right eigenvectors. This is the informal reason why they would be uniformly distributed on the sphere.

  • However, in general with non-zero means, plus some other TBD conditions, one might be able to argue that $\mathbb{E}[x\ x^T]$ has distinct eigenvalues and so the eigenvectors are unambiguous. In this case I would think the right singular vectors of $A$ tend to these unambiguous eigenvectors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.