4
$\begingroup$

I have a question concerning conditional independence. According to wikipedia (yes, maybe not the best source) two random variables are conditionally independent given a third if $$p(x,y|z) = p(x|z)p(y|z)\:\:\forall z.$$

However, I read a mutual information definition of conditional independence that said that two random variables are conditionally independent if $$I[X|Z:Y|Z] = \left\langle\left\langle\log \frac{p(x,y|z)}{p(x|z)p(y|z)}\right\rangle_{X,Y|Z}\right\rangle_Z = 0.$$

Now, I am not sure whether the two equations are equivalent. The first definitely implies the last. However, the last does not imply the first (or does it?), since there could be $z$ for which $p(x,y|z) \not= p(x|z)p(y|z)$ but the set of all $z$ for which that is true has measure zero.

I guess the last equation means conditionally independence only almost surely (in terms of $z$) but not pointwise, but the first requires pointwise equality.

Am I correct? If not, where is my mistake?

$\endgroup$
2
  • $\begingroup$ Conditional Independence needs to be satisfied for every value of the variables involved in the definition. The second definition has to be also satisfied for every $Z$ i.e. point-wise. $\endgroup$
    – Sudarsan
    Oct 11, 2013 at 3:22
  • $\begingroup$ I just realized that I had a mistake in the definition. I am not sure whether I understand you comment. Do you want to say that the second definition has to be satisfied for each $z$ (instead of the mean) in order to be equivalent to the first? $\endgroup$
    – fabee
    Oct 11, 2013 at 6:11

1 Answer 1

3
$\begingroup$

The definition of conditional mutual information should read, in your notations, $$I[X|Z:Y|Z] = \left\langle\left\langle\log \frac{p(x,y|z)}{p(x|z)p(y|z)}\right\rangle_{X,Y\mid Z}\right\rangle_Z, $$ or, equivalently, $$I[X|Z:Y|Z] =\left\langle\log \frac{p(x,y|z)}{p(x|z)p(y|z)}\right\rangle_{X,Y,Z}.$$ For discrete random variables, this is $$I[X|Z:Y|Z] = \sum_zp(z)\sum_{x,y}p(x,y\mid z)\log \frac{p(x,y|z)}{p(x|z)p(y|z)}, $$ or, equivalently, $$I[X|Z:Y|Z] =\sum_{x,y,z}p(x,y,z)\log \frac{p(x,y|z)}{p(x|z)p(y|z)}.$$ Then the same argument than in the unconditional case applies, namely, $$ I[X|Z:Y|Z] = -\sum_{x,y,z}p(x,y,z)\log \frac{p(x|z)p(y|z)}{p(x,y|z)}, $$ and, since $(-\log)$ is convex, $$I[X|Z:Y|Z] \geqslant -\log S,$$ where $$ S=\sum_{x,y,z}p(x,y,z)\frac{p(x|z)p(y|z)}{p(x,y|z)}=\sum_zp(z)\sum_xp(x\mid z)\sum_yp(y\mid z)=\sum_zp(z)\cdot1\cdot1=1, $$ hence $\log S=0$ and $I[X|Z:Y|Z] \geqslant0$. Furthermore, $I[X|Z:Y|Z] =0$ if and only if the convexity inequality above is an equality, that is, the function to be integrated must be constant, that is, here, $$ \frac{p(x|z)p(y|z)}{p(x,y|z)}=c. $$ Summing over $(x,y)$ the identities $p(x|z)p(y|z)=cp(x,y|z)$, one sees that this is only possible with $c=1$ hence $\log S=0$. Finally, $I[X|Z:Y|Z]=0$ if and only if, for every $(x,y,z)$, $$ p(x,y|z)=p(x|z)p(y|z). $$

$\endgroup$
5
  • 1
    $\begingroup$ Thanks for answering (and pointing to the fact, that I still had an error in the definition of conditional mutual information which is corrected now). If I understand you correctly, you argue that we only have equality if we integrate over $\log c$. However, what if $p(x|z)p(y|z)/p(x,y|z)\not=c$ at finitely many points $z$? The integral would still be the same, but we would not have $p(x|z)p(y|z)=p(x,y|z)$ at each $z$. Or am I missing something? $\endgroup$
    – fabee
    Oct 11, 2013 at 9:49
  • $\begingroup$ Rather, the argument is that $E[\log U]=\log E[U]$ implies $U=E[U]$ with full probability. $\endgroup$
    – Did
    Oct 11, 2013 at 10:50
  • 1
    $\begingroup$ I was not expressing myself very clearly. What confuses me is: Say the expectation is taken over a function $f$ of a uniform random variable. $f(u)=1$ for $u\in[0,.5)\cup(.5,1]$ and $f(.5) = e$. Then $E[\log f(U)] = \log E[f(u)]$ but not $f \equiv 1$ everywhere. This might hold with probability one, but not for every $u$. Does that mean that $p(x,y|z)=p(x|z)p(y|z)$ also holds only with probability one? $\endgroup$
    – fabee
    Oct 11, 2013 at 11:26
  • 3
    $\begingroup$ We might hit here the fact that the notations in your post are somewhat non rigorous. I tried to avoid raising this point in my answer... but your formulas confuse the states $x$, $y$, $z$ and the random variables $X$, $Y$, $Z$. The conclusion of my post is that in a discrete setting $p(x,y|z)=p(x|z)p(y|z)$ for every states $x$, $y$, $z$. In the continuous setting, $p(x\mid z)$ has no meaning anyway since $p(\ \mid z)$ becomes a measure (with many different densities, which coincide only almost surely). $\endgroup$
    – Did
    Oct 11, 2013 at 11:33
  • $\begingroup$ That may very well be. I tried being somewhat rigorous by using $X, Y, Z$ for random variables, and $x,y,z$ for actual values of the random variables. However, since I never learned measure theory properly (any good book to recommend?) the notation might not be reasonable. Did I understand you correctly now, that $p(x,y|y)=p(x|z)p(y|z)$ holds only almost surely in the continuous setting? Thank you for spending all the time to solve my issue. $\endgroup$
    – fabee
    Oct 12, 2013 at 14:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .