# mutual information of two normal random vectors

I'm dealing with a question that request to calculate the mutual information of two normal random vectors, this is the description:

If $$\mathbf X\sim \mathcal N(\mu_X,\Sigma_X),~\mathbf Y\sim \mathcal N(\mu_Y,\Sigma_Y)$$ are random vectors with $$N$$ elements, the covariance matrix of $$\mathbf X$$ and $$\mathbf Y,$$ $$\Sigma_{XY}$$ is \begin{align}\Sigma_{XY} &=\mathbb{E}[(\mathbf X-\mathbb{E}\mathbf X)(\mathbf Y-\mathbb{E}\mathbf Y)^\mathsf T]\\ &=\left[\begin{matrix}\Sigma_X&\mathsf{cov}(\mathbf X,\mathbf Y)\\\mathsf{cov}(\mathbf Y,\mathbf X)&\Sigma_Y\end{matrix}\right]\\ &=\left[\begin{matrix}\Sigma_X&\boldsymbol \rho\sqrt{|\Sigma_X||\Sigma_Y|}\\\boldsymbol \rho^\mathsf T\sqrt{|\Sigma_Y||\Sigma_X|}&\Sigma_Y\end{matrix}\right] \end{align}if we define correlation matrix $$\boldsymbol \rho={{\sf cov}(\mathbf X,\mathbf Y)\over \sqrt{|\Sigma_X||\Sigma_Y|}}.$$

Now the PDF of joint distribution $$(\mathbf X,\mathbf Y)$$ is\begin{align} p_{XY}(\mathbf x,\mathbf y) =&\frac1{\sqrt{(2\pi)^{2N}|\Sigma_{XY}|}}\exp\left(-\frac12\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]^\mathsf T\Sigma_{XY}^{-1}\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]\right)\\ =&\frac1{\sqrt{(2\pi)^{2N}(1-|\boldsymbol \rho|^2)|\Sigma_{X}||\Sigma_Y|}}\\ &\times\exp\left(-\frac12\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]^\mathsf T\left[\begin{matrix}\Sigma_X&\boldsymbol \rho\sqrt{|\Sigma_X||\Sigma_Y|}\\\boldsymbol \rho^\mathsf T\sqrt{|\Sigma_X||\Sigma_Y|}&\Sigma_Y\end{matrix}\right]^{-1}\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]\right). \end{align} Also recall the PDF of $$\mathbf X$$ and $$\mathbf Y$$ are \eqalign{ p_X(\mathbf x)=\frac1{\sqrt{(2\pi)^N|\Sigma_X|}}\exp\left(-\frac12(\mathbf x-\mu_X)^\mathsf T\Sigma_X^{-1}(\mathbf x-\mu_X)\right),\\ p_Y(\mathbf y)=\frac1{\sqrt{(2\pi)^N|\Sigma_Y|}}\exp\left(-\frac12(\mathbf y-\mu_Y)^\mathsf T\Sigma_Y^{-1}(\mathbf y-\mu_Y)\right). }

The question is, I want to prove that the mutual information of $$(\mathbf X,\mathbf Y)$$ is \eqalign{ \mathbb I(\mathbf X,\mathbf Y)&=\int_\mathbf x\int_\mathbf yp_{XY}(\mathbf x,\mathbf y)\log\left(p_{XY}(\mathbf x,\mathbf y)\over p_X(\mathbf x)p_Y(\mathbf y)\right)\mathrm d\mathbf x\mathrm d\mathbf y\\ &=-\frac12\log\left(1-|\rho|^2\right). }

Here is my work update to now: If we denote the exponent parts of $$p_{XY}$$'s, $$p_X$$'s and $$p_Y$$'s by $$q_{XY},~q_X$$ and $$q_Y,$$ for simplicity respectively, and coefficient parts by $$1/a_{XY},~1/a_X,$$ and $$1/a_Y.$$

First, \begin{align} &\log\left(p_{XY}(\mathbf x,\mathbf y)\over p_X(\mathbf x)p_Y(\mathbf y)\right)\\ =&\log\left(a^{-1}_{XY}\over a^{-1}_Xa^{-1}_Y\right)+q_{XY}-q_X-q_Y\\ =&\log\left(\sqrt{(2\pi)^N|\Sigma_X|}\sqrt{(2\pi)^N|\Sigma_Y|}\over\sqrt{(2\pi)^{2N}(1-|\boldsymbol \rho|^2)|\Sigma_{X}||\Sigma_Y|} \right)+q_{XY}-q_X-q_Y\\ =&-\frac12\log\left(1-|\rho|^2\right)+q_{XY}-q_X-q_Y.\\ {\rm and}\\ &\int_\mathbf x\int_\mathbf yp_{XY}(\mathbf x,\mathbf y)\left[-\frac12\log\left(1-|\rho|^2\right)\right]\mathrm d\mathbf x\mathrm d\mathbf y=-\frac12\log\left(1-|\rho|^2\right). \end{align} Second, \begin{align} &-2\times (q_{XY}-q_X-q_Y)\\ =&\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]^\mathsf T\Sigma_{XY}^{-1}\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]\\ &-(\mathbf x-\mu_X)^\mathsf T\Sigma_X^{-1}(\mathbf x-\mu_X)-(\mathbf y-\mu_Y)^\mathsf T\Sigma_Y^{-1}(\mathbf y-\mu_Y)\\ =&\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]^\mathsf T\Sigma_{XY}^{-1}\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]\\ &-\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]^\mathsf T\left[\begin{matrix}\Sigma_X&\mathbf 0\\\mathbf 0&\Sigma_Y\end{matrix}\right]^{-1}\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]\\ =&\left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right]^\mathsf T \left\{\Sigma_{XY}^{-1}-\left[\begin{matrix}\Sigma_X&\mathbf 0\\\mathbf 0&\Sigma_Y\end{matrix}\right]^{-1}\right\} \left[\begin{matrix}\mathbf x-\mu_X\\\mathbf y-\mu_Y\end{matrix}\right] \end{align} We need to deal with $$\int\int p_{XY}(q_{XY}-q_X-q_Y),$$ but it's too hard for me after trying many methods.

Can you help me? thanks a lot!

Claim #1: If $$X \sim \mathcal{N}(\mu, \Sigma)$$, where $$\mu \in \mathbb{R}^d, \Sigma \in \mathbb{R}^{d \times d}$$ then $$\mathbb{E}[(X - \mu)^T \Sigma^{-1}(X - \mu)] = d$$ Indeed, the affine transformation $$Y = \Sigma^{-1/2}(X - \mu)$$ gives $$Y \sim \mathcal{N}(0, I_d)$$, where $$I_d$$ is $$d\times d$$ identity matrix. Then, $$(X - \mu)^T \Sigma^{-1}(X - \mu) = \sum_{k = 1}^d Y_k^2 \sim \chi^2(p)$$ (notice that $$Y_k^2 \sim \chi^2(1)$$). Therefore, $$\mathbb{E}[(X - \mu)^T \Sigma^{-1}(X - \mu)] = d$$

Now, $$\int_{\mathbb{R^N}} \int_{\mathbb{R}^N} p_{XY}(x, y)q_{XY}(x, y)dxdy = -\dfrac{1}{2}\mathbb{E}[(Z - \mu')^T(\Sigma')^{-1}(Z - \mu')] = -N$$ where $$Z = (X, Y) \sim \mathcal{N}(\mu', \Sigma')$$

Finally, $$\int_{\mathbb{R^N}} \int_{\mathbb{R}^N} p_{XY}(x, y)q_{X}(x)dxdy = \int_{\mathbb{R}^N} p_X(x)q_X(x)dx = -\dfrac{1}{2}\mathbb{E}[(X - \mu_X)^T\Sigma_X^{-1}(X - \mu)] = -N/2$$ Similarly, $$\int_{\mathbb{R^N}} \int_{\mathbb{R}^N} p_{XY}(x, y)q_{Y}(y)dxdy = \int_{\mathbb{R}^N} p_Y(y)q_Y(y)dy = -\dfrac{1}{2}\mathbb{E}[(Y - \mu_Y)^T\Sigma_Y^{-1}(Y - \mu)] = -N/2$$

Thus, $$\int_{\mathbb{R}^N} \int_{\mathbb{R}^N} p_{XY}(q_{XY} - q_X - q_Y)dxdy = 0$$

• Very very very thankful! Your answer is intelligent and concise! Commented Mar 12 at 23:03