I have a question concerning conditional independence. According to wikipedia (yes, maybe not the best source) two random variables are conditionally independent given a third if $$p(x,y|z) = p(x|z)p(y|z)\:\:\forall z.$$

However, I read a mutual information definition of conditional independence that said that two random variables are conditionally independent if $$I[X|Z:Y|Z] = \left\langle\left\langle\log \frac{p(x,y|z)}{p(x|z)p(y|z)}\right\rangle_{X,Y|Z}\right\rangle_Z = 0.$$

Now, I am not sure whether the two equations are equivalent. The first definitely implies the last. However, the last does not imply the first (or does it?), since there could be $z$ for which $p(x,y|z) \not= p(x|z)p(y|z)$ but the set of all $z$ for which that is true has measure zero.

I guess the last equation means conditionally independence only almost surely (in terms of $z$) but not pointwise, but the first requires pointwise equality.

Am I correct? If not, where is my mistake?

  • $\begingroup$ Conditional Independence needs to be satisfied for every value of the variables involved in the definition. The second definition has to be also satisfied for every $Z$ i.e. point-wise. $\endgroup$ – Sudarsan Oct 11 '13 at 3:22
  • $\begingroup$ I just realized that I had a mistake in the definition. I am not sure whether I understand you comment. Do you want to say that the second definition has to be satisfied for each $z$ (instead of the mean) in order to be equivalent to the first? $\endgroup$ – fabee Oct 11 '13 at 6:11

The definition of conditional mutual information should read, in your notations, $$I[X|Z:Y|Z] = \left\langle\left\langle\log \frac{p(x,y|z)}{p(x|z)p(y|z)}\right\rangle_{X,Y\mid Z}\right\rangle_Z, $$ or, equivalently, $$I[X|Z:Y|Z] =\left\langle\log \frac{p(x,y|z)}{p(x|z)p(y|z)}\right\rangle_{X,Y,Z}.$$ For discrete random variables, this is $$I[X|Z:Y|Z] = \sum_zp(z)\sum_{x,y}p(x,y\mid z)\log \frac{p(x,y|z)}{p(x|z)p(y|z)}, $$ or, equivalently, $$I[X|Z:Y|Z] =\sum_{x,y,z}p(x,y,z)\log \frac{p(x,y|z)}{p(x|z)p(y|z)}.$$ Then the same argument than in the unconditional case applies, namely, $$ I[X|Z:Y|Z] = -\sum_{x,y,z}p(x,y,z)\log \frac{p(x|z)p(y|z)}{p(x,y|z)}, $$ and, since $(-\log)$ is convex, $$I[X|Z:Y|Z] \geqslant -\log S,$$ where $$ S=\sum_{x,y,z}p(x,y,z)\frac{p(x|z)p(y|z)}{p(x,y|z)}=\sum_zp(z)\sum_xp(x\mid z)\sum_yp(y\mid z)=\sum_zp(z)\cdot1\cdot1=1, $$ hence $\log S=0$ and $I[X|Z:Y|Z] \geqslant0$. Furthermore, $I[X|Z:Y|Z] =0$ if and only if the convexity inequality above is an equality, that is, the function to be integrated must be constant, that is, here, $$ \frac{p(x|z)p(y|z)}{p(x,y|z)}=c. $$ Summing over $(x,y)$ the identities $p(x|z)p(y|z)=cp(x,y|z)$, one sees that this is only possible with $c=1$ hence $\log S=0$. Finally, $I[X|Z:Y|Z]=0$ if and only if, for every $(x,y,z)$, $$ p(x,y|z)=p(x|z)p(y|z). $$

  • 1
    $\begingroup$ Thanks for answering (and pointing to the fact, that I still had an error in the definition of conditional mutual information which is corrected now). If I understand you correctly, you argue that we only have equality if we integrate over $\log c$. However, what if $p(x|z)p(y|z)/p(x,y|z)\not=c$ at finitely many points $z$? The integral would still be the same, but we would not have $p(x|z)p(y|z)=p(x,y|z)$ at each $z$. Or am I missing something? $\endgroup$ – fabee Oct 11 '13 at 9:49
  • $\begingroup$ Rather, the argument is that $E[\log U]=\log E[U]$ implies $U=E[U]$ with full probability. $\endgroup$ – Did Oct 11 '13 at 10:50
  • 1
    $\begingroup$ I was not expressing myself very clearly. What confuses me is: Say the expectation is taken over a function $f$ of a uniform random variable. $f(u)=1$ for $u\in[0,.5)\cup(.5,1]$ and $f(.5) = e$. Then $E[\log f(U)] = \log E[f(u)]$ but not $f \equiv 1$ everywhere. This might hold with probability one, but not for every $u$. Does that mean that $p(x,y|z)=p(x|z)p(y|z)$ also holds only with probability one? $\endgroup$ – fabee Oct 11 '13 at 11:26
  • 2
    $\begingroup$ We might hit here the fact that the notations in your post are somewhat non rigorous. I tried to avoid raising this point in my answer... but your formulas confuse the states $x$, $y$, $z$ and the random variables $X$, $Y$, $Z$. The conclusion of my post is that in a discrete setting $p(x,y|z)=p(x|z)p(y|z)$ for every states $x$, $y$, $z$. In the continuous setting, $p(x\mid z)$ has no meaning anyway since $p(\ \mid z)$ becomes a measure (with many different densities, which coincide only almost surely). $\endgroup$ – Did Oct 11 '13 at 11:33
  • $\begingroup$ That may very well be. I tried being somewhat rigorous by using $X, Y, Z$ for random variables, and $x,y,z$ for actual values of the random variables. However, since I never learned measure theory properly (any good book to recommend?) the notation might not be reasonable. Did I understand you correctly now, that $p(x,y|y)=p(x|z)p(y|z)$ holds only almost surely in the continuous setting? Thank you for spending all the time to solve my issue. $\endgroup$ – fabee Oct 12 '13 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.