# Decomposition of mutual information for conditionally independent variables

I have a question regarding the mutual information of conditionally independent random variables (observations).

Given $p(x,y|z) = p(x|z)p(y|z)$ where $z$ corresponds to a latent variable, I was wondering if an established approach exists for the decomposition of the mutual information $I(x;y)$ such that only quantities (MI / entropy / etc.) between one variable and the latent variable need to be calculated $I(x;y) = F(I(x;z), I(y;z))$?

• Are you by any chance looking for the conditional mutual information $I(X;Y\mid Z)$? Per your problem statement: given $Z$; the r.v.'s $X$ and $Y$ are independent, so conditional MI is zero. If value of $Z$ is revealed, knowing $Y$ provides no further information on $X$. – mikkola Nov 10 '15 at 6:54
• That's true that the conditional mutual information will be zero for the current setting, and this motivates my pursuit of this decomposition. Essentially, knowing the value of $Y$ reduces the uncertainty in the value of $X$, but this is a consequence of the dependency on the latent variable $Z$. As all "information" is contained in the latent variable, I was hoping to be able to express the MI $I(X;Y)$ indirectly, using for example MI with the latent variable ($I(X;Z), I(Y;Z)$) – Peter Nov 10 '15 at 9:38

## 1 Answer

To my knowledge, there is no such general relation, but if $$(x, y, z)\in\mathbb{R}^3$$ follows a jointly Gaussian distribution and x and y are conditionally independent on z, then there is a formula $$I(x,y)=f(I(x,z), I(y,z))$$.

The relation is found in the conference proceeding Sensor selection in high-dimensional Gaussian trees with nuisances (NIPS 2013), which tells you

$$\log\left(1-\exp(-2I(x,y))\right)=\log\left(1-\exp(-2I(x,z))\right)+\log\left(1-\exp(-2I(y,z))\right)$$

The formula can be extended by mathematical induction to a larger Gaussian tree.