Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

How do we derive the following: $$ -\sum_{x} \sum_{y} p(x,y) \left( \log_{2} p(x) + \log_{2} p(y) - \log_{2} p(x,y) \right)$$ $$= -\sum_{y} p(y) \log_{2} p(y)+\sum_{x} p(x) \sum_{y} p(y|x) \log_{2} p(y|x)$$

The first equation simplified is: $$-\sum_{x} \sum_{y} p(x,y) \log_{2} p(x)-\sum_{x}\sum_{y} p(x,y) \log_{2} p(x,y)+\sum_{x}\sum_{y} p(x,y) \log_{2} p(x,y)$$

Do we just use the fact that $p(x,y) = p(y)p(x|y) = p(x)p(y|x)$ and substitute in?

share|cite|improve this question
What do you get after doing the substitution? – Alex R. Feb 1 '13 at 0:15

This is just a guess: We know that $$-\sum_{x} \sum_{y} p(x,y) \log_{2} p(y) = -\sum_{y} \sum_{x} p(x,y) \log_{2} p(y)$$

$$= -\sum_{y} p(y) \log_{2} p(y)$$

Also $$\sum_{x} \sum_{y} p(x,y) \log_{2}p(x,y)- \sum_{x} \sum_{y} p(x,y) \log_{2} p(x)$$ $$ = \sum_{x} \sum_{y} p(x,y) \log_{2} \frac{p(x,y)}{p(x)}$$

$$ = \sum_{x} \sum_{y} p(y|x) p(x) \log_{2} p(y|x)$$

$$= \sum_{x} p(x) \sum_{y} p(y|x) \log_{2} p(y|x)$$

Thus $$ -\sum_{x} \sum_{y} p(x,y) \left( \log_{2} p(x) + \log_{2} p(y) - \log_{2} p(x,y) \right)$$ $$= -\sum_{y} p(y) \log_{2} p(y)+\sum_{x} p(x) \sum_{y} p(y|x) \log_{2} p(y|x)$$

share|cite|improve this answer
The first thing that We know in your post seems wrong. – Did Feb 1 '13 at 16:35
@Did: What is wrong about it? – proton Feb 1 '13 at 16:36
Try some concrete cases. You seem to have confused $\log p(x,y)$ with $\log p(y)$. – Did Feb 1 '13 at 16:45
@Did: I was using the fact that $\sum_{x} p(x,y) = p(y)$. Then $\sum_{x} p(x,y) \log_{2} p(x,y) = p(y) \log_{2} p(y)$. – proton Feb 1 '13 at 16:51
No. As I already indicated, this would be true if one had $\log p(y)$ instead of $\log p(x,y)$ in the summation over $(x,y)$. – Did Feb 1 '13 at 17:03

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.