1
$\begingroup$

I'm dealing with a question that request to calculate the mutual information of two normal random vectors, this is the description:

If $\mathbf X\sim \mathcal N(\mu_X,\Sigma_X),~\mathbf Y\sim \mathcal N(\mu_Y,\Sigma_Y)$ are random vectors with $N$ elements, the covariance matrix of $\mathbf X$ and $\mathbf Y,$ $\Sigma_{XY}$ is $$\begin{align}\Sigma_{XY} &=\mathbb{E}[(\mathbf X-\mathbb{E}\mathbf X)(\mathbf Y-\mathbb{E}\mathbf Y)^\mathsf T]\\ &=\left[\begin{matrix}\Sigma_X&\mathsf{cov}(\mathbf X,\mathbf Y)\\\mathsf{cov}(\mathbf Y,\mathbf X)&\Sigma_Y\end{matrix}\right]\\ &=\left[\begin{matrix}\Sigma_X&\boldsymbol \rho\sqrt{|\Sigma_X||\Sigma_Y|}\\\boldsymbol \rho^\mathsf T\sqrt{|\Sigma_Y||\Sigma_X|}&\Sigma_Y\end{matrix}\right] \end{align}$$if we define correlation matrix $$\boldsymbol \rho={{\sf cov}(\mathbf X,\mathbf Y)\over \sqrt{|\Sigma_X||\Sigma_Y|}}.$$

Now the PDF of joint distribution $(\mathbf X,\mathbf Y)$ is$$\begin{align} p_{XY}(\mathbf x,\mathbf y) =&\frac1{\sqrt{(2\pi)^{2N}|\Sigma_{XY}|}}\exp\left(-\frac12\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]^\mathsf T\Sigma_{XY}^{-1}\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]\right)\\ =&\frac1{\sqrt{(2\pi)^{2N}(1-|\boldsymbol \rho|^2)|\Sigma_{X}||\Sigma_Y|}}\\ &\times\exp\left(-\frac12\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]^\mathsf T\left[\begin{matrix}\Sigma_X&\boldsymbol \rho\sqrt{|\Sigma_X||\Sigma_Y|}\\\boldsymbol \rho^\mathsf T\sqrt{|\Sigma_X||\Sigma_Y|}&\Sigma_Y\end{matrix}\right]^{-1}\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]\right). \end{align}$$ Also recall the PDF of $\mathbf X$ and $\mathbf Y$ are $$\eqalign{ p_X(\mathbf x)=\frac1{\sqrt{(2\pi)^N|\Sigma_X|}}\exp\left(-\frac12(\mathbf x-\mu_X)^\mathsf T\Sigma_X^{-1}(\mathbf x-\mu_X)\right),\\ p_Y(\mathbf y)=\frac1{\sqrt{(2\pi)^N|\Sigma_Y|}}\exp\left(-\frac12(\mathbf y-\mu_Y)^\mathsf T\Sigma_Y^{-1}(\mathbf y-\mu_Y)\right). }$$

The question is, I want to prove that the mutual information of $(\mathbf X,\mathbf Y)$ is $$\eqalign{ \mathbb I(\mathbf X,\mathbf Y)&=\int_\mathbf x\int_\mathbf yp_{XY}(\mathbf x,\mathbf y)\log\left(p_{XY}(\mathbf x,\mathbf y)\over p_X(\mathbf x)p_Y(\mathbf y)\right)\mathrm d\mathbf x\mathrm d\mathbf y\\ &=-\frac12\log\left(1-|\rho|^2\right). }$$

Here is my work update to now: If we denote the exponent parts of $p_{XY}$'s, $p_X$'s and $p_Y$'s by $q_{XY},~q_X$ and $q_Y,$ for simplicity respectively, and coefficient parts by $1/a_{XY},~1/a_X,$ and $1/a_Y.$

First, $$\begin{align} &\log\left(p_{XY}(\mathbf x,\mathbf y)\over p_X(\mathbf x)p_Y(\mathbf y)\right)\\ =&\log\left(a^{-1}_{XY}\over a^{-1}_Xa^{-1}_Y\right)+q_{XY}-q_X-q_Y\\ =&\log\left(\sqrt{(2\pi)^N|\Sigma_X|}\sqrt{(2\pi)^N|\Sigma_Y|}\over\sqrt{(2\pi)^{2N}(1-|\boldsymbol \rho|^2)|\Sigma_{X}||\Sigma_Y|} \right)+q_{XY}-q_X-q_Y\\ =&-\frac12\log\left(1-|\rho|^2\right)+q_{XY}-q_X-q_Y.\\ {\rm and}\\ &\int_\mathbf x\int_\mathbf yp_{XY}(\mathbf x,\mathbf y)\left[-\frac12\log\left(1-|\rho|^2\right)\right]\mathrm d\mathbf x\mathrm d\mathbf y=-\frac12\log\left(1-|\rho|^2\right). \end{align}$$ Second, $$\begin{align} &-2\times (q_{XY}-q_X-q_Y)\\ =&\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]^\mathsf T\Sigma_{XY}^{-1}\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]\\ &-(\mathbf x-\mu_X)^\mathsf T\Sigma_X^{-1}(\mathbf x-\mu_X)-(\mathbf y-\mu_Y)^\mathsf T\Sigma_Y^{-1}(\mathbf y-\mu_Y)\\ =&\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]^\mathsf T\Sigma_{XY}^{-1}\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]\\ &-\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]^\mathsf T\left[\begin{matrix}\Sigma_X&\mathbf 0\\\mathbf 0&\Sigma_Y\end{matrix}\right]^{-1}\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]\\ =&\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]^\mathsf T \left\{\Sigma_{XY}^{-1}-\left[\begin{matrix}\Sigma_X&\mathbf 0\\\mathbf 0&\Sigma_Y\end{matrix}\right]^{-1}\right\} \left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right] \end{align}$$ We need to deal with $\int\int p_{XY}(q_{XY}-q_X-q_Y),$ but it's too hard for me after trying many methods.

Can you help me? thanks a lot!

$\endgroup$

1 Answer 1

1
$\begingroup$

Claim #1: If $X \sim \mathcal{N}(\mu, \Sigma)$, where $\mu \in \mathbb{R}^d, \Sigma \in \mathbb{R}^{d \times d}$ then $$ \mathbb{E}[(X - \mu)^T \Sigma^{-1}(X - \mu)] = d $$ Indeed, the affine transformation $Y = \Sigma^{-1/2}(X - \mu)$ gives $Y \sim \mathcal{N}(0, I_d)$, where $I_d$ is $d\times d$ identity matrix. Then, $$ (X - \mu)^T \Sigma^{-1}(X - \mu) = \sum_{k = 1}^d Y_k^2 \sim \chi^2(p) $$ (notice that $Y_k^2 \sim \chi^2(1)$). Therefore, $\mathbb{E}[(X - \mu)^T \Sigma^{-1}(X - \mu)] = d$


Now, $$ \int_{\mathbb{R^N}} \int_{\mathbb{R}^N} p_{XY}(x, y)q_{XY}(x, y)dxdy = -\dfrac{1}{2}\mathbb{E}[(Z - \mu')^T(\Sigma')^{-1}(Z - \mu')] = -N $$ where $Z = (X, Y) \sim \mathcal{N}(\mu', \Sigma')$

Finally, $$ \int_{\mathbb{R^N}} \int_{\mathbb{R}^N} p_{XY}(x, y)q_{X}(x)dxdy = \int_{\mathbb{R}^N} p_X(x)q_X(x)dx = -\dfrac{1}{2}\mathbb{E}[(X - \mu_X)^T\Sigma_X^{-1}(X - \mu)] = -N/2 $$ Similarly, $$ \int_{\mathbb{R^N}} \int_{\mathbb{R}^N} p_{XY}(x, y)q_{Y}(y)dxdy = \int_{\mathbb{R}^N} p_Y(y)q_Y(y)dy = -\dfrac{1}{2}\mathbb{E}[(Y - \mu_Y)^T\Sigma_Y^{-1}(Y - \mu)] = -N/2 $$

Thus, $$ \int_{\mathbb{R}^N} \int_{\mathbb{R}^N} p_{XY}(q_{XY} - q_X - q_Y)dxdy = 0 $$

$\endgroup$
1
  • 1
    $\begingroup$ Very very very thankful! Your answer is intelligent and concise! $\endgroup$ Commented Mar 12 at 23:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .