I am learning about mutual information, and am confused about one of the definitions. Mutual information is defined as $ I(X;Y) = H(X) - H(X | Y) $


$$ H(X) = \sum_{x} p(x) \log \frac{1}{p(x)} ,$$

and similarly,

$$ H(X|Y) = \sum_{x,y} p(x,y) \log \frac{1}{p(x|y)} $$

where $H(X)$ is concerned we can then say that,

$$ \begin{align*} \sum_x p(x) \log \frac{1}{p(x)} &= \sum_x \left( (p(x) \log \frac{1}{p(x)}) \sum_y p(y|x) \right) \\ &= \sum_{x,y} p(x)p(y|x) \log \frac{1}{p(x)} = \\ &= \sum_{x,y} p(x,y) \log \frac{1}{p(x)} \end{align*} $$

because $ \sum \limits_y p(y|x) = 1 $ for any $x$.

I believe this is how the derivation is supposed to go, and combined with $H(X|Y)$ eventually leads to the canonical equation,

$$\sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)p(y)} .$$

But it seems to me that it is equally true to say that,

$$ \begin{align*} \sum_{x} p(x) \log \frac{1}{p(x)} &= \sum_{x} \left\{ ( p(x) \log \frac{1}{p(x)} ) \sum_{y} p(y) \right\} \\ &= \sum_{x,y} p(x)p(y) \log \frac{1}{p(x)} \end{align*} $$

because we also have $ \sum_{y} p(y) = 1 $ by definition.

The problem I'm having is that this latter version implies that,

$$\sum_{x,y} p(x,y) \log \frac{1}{p(x)} = \sum_{x,y} p(x)p(y) \log \frac{1}{p(x)} $$

which implies that $ p(x,y) = p(x)p(y) $ which also implies that $X$ and $Y$ are independent. I know this last conclusion is false, because it means there would never be any mutual information and thus that this metric would be pointless, but I can't figure out where I'm going wrong. It would be great to have someone point out the mistake I am making in the latter case.


1 Answer 1


Your formula $$\sum_{x,y}p(x,y)\log \frac{1}{p(x)} = \sum_{x,y}p(x)p(y)\log \frac{1}{p(x)}$$ is indeed correct, since you can always average out $y$ because $p(x)$ does obviously not depend on $y$. Howvever, it does not imply that $p(x,y) = p(x)p(y)$ ($2 -2 = 1 -1 = 0$, but not $1=2$).

  • $\begingroup$ Oh geeez... I'm not sure whether to be happy or feel even more retarded. In any case thanks a bunch for pointing out what should have been obvious. I've marked your answer as correct, but I don't have enough reputation to upvote you. $\endgroup$
    – user20314
    Commented Nov 29, 2011 at 11:30
  • $\begingroup$ I so know the feeling :). In any case, you're welcome. Happy to be of help. $\endgroup$
    – fabee
    Commented Nov 29, 2011 at 19:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .