Reference request: Convergence of singular values of tall random matrix

Question

Let $X_n\in\mathbb{R}^{n\times m}$ be a matrix whose entries are i.i.d. random variables with zero mean and variance $\sigma^2$. Let $m$ be a fixed integer and $\|\cdot\|$ denote the 2-norm of a matrix. I would like to prove that $$ \left\|\frac{1}{n} X_n^\top X_n -\sigma^2 I\right\| \overset{P}{\to} 0 \ \ \ \text{ as }\ \ \ n\to\infty $$ where $\overset{P}{\to} 0$ denotes convergence in probability.

I believe that this should be a well-known fact, which should follow from the fact that high-dimensional random vectors are almost orthogonal as their dimension increases (see e.g. here). However, I couldn't find a reference with a rigorous proof. Hence, I would really appreciate any comment with pointers to the literature. Thanks!

Answer 1

In fact, since we are already assuming i.i.d. sequence, it is a direct consequence of the famous Strong (Weak) law of large numbers. (We don't need any advanced theorems other than SLLN.) The proof goes like this. Let $X_{k,j}$ $(k=1,2,\ldots, n$ ,$j=1,2,\ldots m)$ denote $(k,j)$-th entry of $X_n$. Observe that $(i,j)$-th entry of the matrix $\frac{1}{n}X_n^\top X_n$ is given by $$ \frac{1}{n}\sum_{k=1}^n X_{ki}X_{kj}. $$ Note that for each $(i,j)$, $(X_{ki}X_{kj})_{k\in\Bbb N}$ is an i.i.d. sequence of random variables. Thus by Strong law of large numbers, we have $$\frac{1}{n}\sum_{k=1}^n X_{ki}X_{kj}\to_{\text{a.s.}} E[X_{1i}X_{1j}]=\begin{cases}\sigma^2,\quad i=j\\0,\quad i\ne j\end{cases}. $$ This implies that each entry of $m\times m$ matrix $M_n:=\frac{1}{n}X_n^\top X_n-\sigma^2I$ converges to $0$ almost surely. Now, since $$ \|M_n\|_2^2= \sum_{i,j=1}^m |M_{i,j}|^2, $$ it follows that $$ \lim_{n\to\infty}\|M_n\|_2=0 $$ almost surely. It follows that $$ \|\frac{1}{n}X_n^\top X_n-\sigma^2I\|_2\to_p 0, $$ since almost sure convergence implies convergence in probability.