7
$\begingroup$

Let $Z \sim \mu_\lambda$ be a the Marchenko-Pastur law with parameter $\lambda \in (0,\infty)$, and let $k$ be a negative integer

Question. Is there an analytic formula the $k$th moment for $m_k(\lambda) = \mathbb E[Z^k]$ ?

Note. I'm particularly inteteresting in $m_{-1}(\lambda)$ and $m_{-2}(\lambda)$.

Motivation

$m_k(\lambda)$ is the trace of the pseudo-inverse of the $k$th power a Wishart random matrix (inverse covariance matrix in gaussian iid random design).

Application: generalization error of least-squares regression

Consider a distribution $P$ on $\mathbb R^d \times \mathbb R$ defined by $(x,y) \sim P$ iff $x \sim N(0,(1/d)I_d)$ and $y|x \sim N(w_\star^\top x,\sigma^2)$, where $w_\star \in \mathbb R^d$ and $\sigma \ge 0$ are fixed by unknown. Thus a point drawn from $P$ is of the form $(x,y)$ where $y=xw_\star+\eta$, with $\eta \sim N(0,\sigma^2)$.

Let $\mathcal D_n := \{(x_1,y_1),\ldots,(x_n,y_n)\} \sim P^n$ be an iid sample from $P$. Consider the problem of estimating $w_\star$ from the data $\mathcal D_n$. For $n < d$, $XX^\top$ is invertible w.p $1$ and the least-squares solution is given by $\hat{w} = X^\top(XX^\top)^{-1}y=P_X w_\star + X(XX^\top)^{-1}\varepsilon$, where

  • $X$ is the $n \times d$ matrix with $i$th row $x_i$,
  • $P_X := X^\top (XX^\top)^{-1} X$ is the orthogonal projection matrix onto the row space of $X$,
  • $\epsilon$ is a column vector in $\mathbb R^n$ with iid components from $N(0,\sigma^2 I)$, and
  • $y = Xw_\star+\varepsilon$.

For any $w \in \mathbb R^d$, let $f_w:\mathbb R^d \to \mathbb R$ be the linear function $f_w(x):=w_\star^\top x$. Thus, the generalization error of the model $f_{\hat{w}}$ is given by

$$ \begin{split} E_g &:= \mathbb E_{x}\mathbb E_\varepsilon[(w_\star^\top P_X x + \varepsilon^\top (XX^\top)^{-1} Xx - w_\star^\top x)^2]\\ &= (1/d)\|(I-P_X)w_\star\|^2+(\sigma^2/d)\mbox{tr}((XX^\top)^{-1}). \end{split} $$

Noise only model. For simplicity, assume $w_\star = 0$. Then, in the limit when $n,d \to \infty$ with $n/d \to \lambda \in [0,1)$, we have $$ E_g = \sigma^2\frac{1}{d}\mbox{tr}((XX^\top)^{-1}) \to \sigma^2 m_{-1}(\lambda),\,X\text{-a.s}. $$

Thanks to the accepted answer, we conclude that $E_g \to \dfrac{\sigma^2}{1-\lambda}$, $X$-a.s.

$\endgroup$
2
  • 1
    $\begingroup$ I just found a nice blog which computes the positive moments of the Marchenko-Pastur distribution, using that the moments of the semicircle law are the Catalan numbers: djalil.chafai.net/blog/2011/01/29/the-marchenko-pastur-law $\endgroup$
    – charmd
    Sep 5, 2021 at 14:45
  • $\begingroup$ @charmd Yes, I also stumbled on that site. Nice blog indeed. $\endgroup$
    – dohmatob
    Sep 21, 2021 at 11:17

1 Answer 1

8
$\begingroup$

My answer will merely rely on the following reference (that I found here). It is an interesting big list of integrals, that you can read there: I.S. Gradshteyn, I.M. Ryzhik, Table of integrals, series, and products.

Denoting $R = (\lambda^+-x)(x-\lambda^-)$, you're interested in $\displaystyle{\int} \frac{\sqrt{R}}{2\pi\lambda x \cdot x^m}dx$ for $m \ge 1$. You can check that you have the following equalities (see 2.265):

  • for $m \ge 2$, $\displaystyle{\int_{\lambda^-}^{\lambda^+}} \frac{\sqrt{R}}{x^m}dx = \frac{(2m-5)(\lambda^+ + \lambda^-)}{2(m-1)\lambda^+\lambda^-} \displaystyle{\int_{\lambda^-}^{\lambda^+}} \frac{\sqrt{R}}{x^{m-1}}dx - \frac{m-4}{(m-1)\lambda^+\lambda^-}\displaystyle{\int_{\lambda^-}^{\lambda^+}} \frac{R}{x^{m-2}}dx$

  • for $m=1$, $\displaystyle{\int_{\lambda^-}^{\lambda^+}} \frac{\sqrt{R}}{x}dx = 2 \pi \lambda$.

  • for $m=0$, $\displaystyle{\int_{\lambda^-}^{\lambda^+}} \sqrt{R}dx = 2 \pi \lambda$.

Thus, $m_1(\lambda) = m_0(\lambda)=1$, and for $k \ge 1$, we have the recurrence relation $$m_{-k}(\lambda) = \frac{1}{k(1-\lambda)^2} \Big((2k-3)(1+\lambda) m_{-k+1}(\lambda) - (k-3)m_{-k+2}(\lambda)\Big) \quad (\star)$$

In particular, $m_{-1}(\lambda) = \frac{1}{(1-\lambda)^2} (-(1+\lambda) + 2) = \frac{1}{1-\lambda}$, and $m_{-2}(\lambda) = \frac{1}{(1-\lambda)^3}$.


If you want additional evidence that no mistakes were made, here are the negative moments, computed either by numerical integration, or with the previous recurrence formula: they match.

enter image description here


Three remarks:

  • Note that the formula for $m_{-1}$ and $m_{-2}$ do not generalize (no general power law expression for the moments).

  • I do not know if the previous recurrence relation can lead to a straightforward formula (say, that wouldn't use a sum), but I don't believe so.

  • It should also be possible to derive an exact formula for the antiderivative of $\frac{1}{x^m} f_{MP}(x)$, if that interests you. However, I do not believe it could be made very explicit either.

$\endgroup$
2
  • 1
    $\begingroup$ Thanks for the detailed answer ! It also turns out I was really only interested in $m_{-k}$ for $k \in \{1,2\}$. Out of curiosity for the other cases, I opened another question here math.stackexchange.com/q/4226136/168758. $\endgroup$
    – dohmatob
    Aug 16, 2021 at 22:22
  • $\begingroup$ For $c \in \mathbb R$, define $M_k(\lambda; c) := \mathbb E[(Z+c)^k]$. Note that $m_k(\lambda) = M_k(\lambda;0)$. Would the computation of $M_k(\lambda;c)$ for general $c$ be much more difficult from that of $c=0$ (done in your answer) ? $\endgroup$
    – dohmatob
    Sep 21, 2021 at 11:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .