I've been having trouble figuring out how to directly prove that the covariance of the multivariate Gaussian distribution $p(x) = \dfrac{1}{{{(2\pi)}^{\frac{n}{2}}}|\Sigma|^{\frac{1}{2}}}\exp\{{-\dfrac{1}{2}(x - \mu)^T \Sigma^{-1}(x-\mu)}\}$ where $x \in \mathbb{R}^n$ and $\Sigma$ is positive definite, is actually $\Sigma$.

That is to say, given $\Sigma$ and the above density function, I want to prove that the covariance is $\Sigma$. I've been able to prove that the mean is $\mu$ by diagonalizing the matrix $\Sigma$ and integrating, but I'm struggling to do it for the proof that the covariance is $\Sigma$. Can anyone help?

  • $\begingroup$ Do you know how to do it in the univariate case? $\endgroup$ – Daniel McLaury Aug 28 '16 at 6:29
  • $\begingroup$ yes. it's the multivariate that confuses me $\endgroup$ – pyrrhic Aug 28 '16 at 7:04

You need to perform a linear change of variables to do this. Let $H$ be a solution to $H^TH=\Sigma^{-1}$ (it exists because $\Sigma$ is positive definite). We will consider the random vector $y=H(x-\mu)$.

We have $y^Ty=(x-\mu)^TH^TH(x-\mu)=(x-\mu)^T\Sigma^{-1}(x-\mu)$ and $|H||\Sigma|^{1/2}=1$. So the pdf of $y$ is given by

$p(y)=\frac1{|H|}p(x)=\frac{1}{|H|}\frac{1}{(2\pi)^{n/2}|\Sigma|^{1/2}}\exp\left(-\frac12(x-\mu)^T\Sigma^{-1}(x-\mu)\right)= \frac{1}{(2\pi)^{n/2}}\exp\left(-\frac12y^Ty\right)$

This means that the entries of $y$ are independent standard normal variables, so $\mathrm{Cov}[y]=I$.

In general we have $\mathrm{Cov}[Ax+b]=A\mathrm{Cov}[X]A^T$ (this is the multivariate version of $\mathrm{Var}(ax+b)=a^2\mathrm{Var}(x)$, and can be shown straightforwardly by considering the definition $\mathrm{Cov}[X]=\mathbb{E}[XX^T]-\mathbb{E}[X]\mathbb{E}[X]^T$).

So in our case $I=\mathrm{Cov}[y]=\mathrm{Cov}[H(x-\mu)]=H\mathrm{Cov}[x]H^T$, and $\mathrm{Cov}[x]=H^{-1}(H^T)^{-1}=(H^TH)^{-1}=\Sigma$.

  • $\begingroup$ I have a relevant question though. Why is sigma positive definite? In general a covariance matrix needs only to be positive-semidefinite right? $\endgroup$ – Zzy1130 Mar 25 '20 at 3:29
  • $\begingroup$ essentially my question is where does that inverse sigma come from $\endgroup$ – Zzy1130 Mar 25 '20 at 4:15
  • 1
    $\begingroup$ @Zzy1130: That's simply an assumption stated explicitly in the question. If the covariance is not positive definite, the pdf is not given by the formula shown in the question (in fact there isn't a pdf in the usual sense), and we have an entirely different problem on our hands, which we can describe and solve separately. $\endgroup$ – Meni Rosenfeld Mar 25 '20 at 14:28
  • $\begingroup$ I see. Thank you so much. $\endgroup$ – Zzy1130 Mar 26 '20 at 9:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.