# Deriving an equation

How do we derive the following: $$-\sum_{x} \sum_{y} p(x,y) \left( \log_{2} p(x) + \log_{2} p(y) - \log_{2} p(x,y) \right)$$ $$= -\sum_{y} p(y) \log_{2} p(y)+\sum_{x} p(x) \sum_{y} p(y|x) \log_{2} p(y|x)$$

The first equation simplified is: $$-\sum_{x} \sum_{y} p(x,y) \log_{2} p(x)-\sum_{x}\sum_{y} p(x,y) \log_{2} p(x,y)+\sum_{x}\sum_{y} p(x,y) \log_{2} p(x,y)$$

Do we just use the fact that $p(x,y) = p(y)p(x|y) = p(x)p(y|x)$ and substitute in?

-
What do you get after doing the substitution? – Alex R. Feb 1 '13 at 0:15

This is just a guess: We know that $$-\sum_{x} \sum_{y} p(x,y) \log_{2} p(y) = -\sum_{y} \sum_{x} p(x,y) \log_{2} p(y)$$

$$= -\sum_{y} p(y) \log_{2} p(y)$$

Also $$\sum_{x} \sum_{y} p(x,y) \log_{2}p(x,y)- \sum_{x} \sum_{y} p(x,y) \log_{2} p(x)$$ $$= \sum_{x} \sum_{y} p(x,y) \log_{2} \frac{p(x,y)}{p(x)}$$

$$= \sum_{x} \sum_{y} p(y|x) p(x) \log_{2} p(y|x)$$

$$= \sum_{x} p(x) \sum_{y} p(y|x) \log_{2} p(y|x)$$

Thus $$-\sum_{x} \sum_{y} p(x,y) \left( \log_{2} p(x) + \log_{2} p(y) - \log_{2} p(x,y) \right)$$ $$= -\sum_{y} p(y) \log_{2} p(y)+\sum_{x} p(x) \sum_{y} p(y|x) \log_{2} p(y|x)$$

-
The first thing that We know in your post seems wrong. – Did Feb 1 '13 at 16:35
@Did: What is wrong about it? – proton Feb 1 '13 at 16:36
Try some concrete cases. You seem to have confused $\log p(x,y)$ with $\log p(y)$. – Did Feb 1 '13 at 16:45
@Did: I was using the fact that $\sum_{x} p(x,y) = p(y)$. Then $\sum_{x} p(x,y) \log_{2} p(x,y) = p(y) \log_{2} p(y)$. – proton Feb 1 '13 at 16:51
No. As I already indicated, this would be true if one had $\log p(y)$ instead of $\log p(x,y)$ in the summation over $(x,y)$. – Did Feb 1 '13 at 17:03