I have a question regarding the mutual information of conditionally independent random variables (observations).

Given $p(x,y|z) = p(x|z)p(y|z)$ where $z$ corresponds to a latent variable, I was wondering if an established approach exists for the decomposition of the mutual information $I(x;y)$ such that only quantities (MI / entropy / etc.) between one variable and the latent variable need to be calculated $I(x;y) = F(I(x;z), I(y;z))$?

  • $\begingroup$ Are you by any chance looking for the conditional mutual information $I(X;Y\mid Z)$? Per your problem statement: given $Z$; the r.v.'s $X$ and $Y$ are independent, so conditional MI is zero. If value of $Z$ is revealed, knowing $Y$ provides no further information on $X$. $\endgroup$ – mikkola Nov 10 '15 at 6:54
  • $\begingroup$ That's true that the conditional mutual information will be zero for the current setting, and this motivates my pursuit of this decomposition. Essentially, knowing the value of $Y$ reduces the uncertainty in the value of $X$, but this is a consequence of the dependency on the latent variable $Z$. As all "information" is contained in the latent variable, I was hoping to be able to express the MI $I(X;Y)$ indirectly, using for example MI with the latent variable ($I(X;Z), I(Y;Z)$) $\endgroup$ – Peter Nov 10 '15 at 9:38

To my knowledge, there is no such general relation, but if $(x, y, z)\in\mathbb{R}^3$ follows a jointly Gaussian distribution and x and y are conditionally independent on z, then there is a formula $I(x,y)=f(I(x,z), I(y,z))$.

The relation is found in the conference proceeding Sensor selection in high-dimensional Gaussian trees with nuisances (NIPS 2013), which tells you


The formula can be extended by mathematical induction to a larger Gaussian tree.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.