Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

According to the Wikipedia article on conditional entropy, $\sum p(x,y)\log p(x)=\sum p(x)\log p(x)$. Can someone please explain how?

share|cite|improve this question
up vote 3 down vote accepted

$$\sum_{x,y} p(x,y)\log p(x)=\sum_x\sum_yp(x,y)\log p(x)=\sum_x\left(\sum_yp(x,y)\right)\log p(x)=\sum_x p(x)\log p(x)$$

share|cite|improve this answer
@joriki- I still dint get it. Can you please give a little more detail? Might be I'm missing something very basic here.. some concept of probability – user23844 Jan 27 '12 at 19:00
@Buxme In the step $\sum \limits_{y} p(x,y) = p(x)$, you could think of the variable $y$ as being marginalised out. – Srivatsan Jan 27 '12 at 19:02
@Buxme: Which of the three steps are you having trouble with? – joriki Jan 27 '12 at 19:42
@joriki- How did y get away? The step Srivatsan mentioned in the second comment.. – user23844 Jan 27 '12 at 19:49
@Buxme: That step is essentially the law of total probability. The wikipedia page contains a detailed explanation. Read it and get back to us if you still don't understand. – Srivatsan Jan 27 '12 at 20:54

In the article $p(x,y)$ is summed (integrated) over all possible values of y for a fixed value of x.

$\sum_{y\in\mathcal Y} p(x,y)= p(x)$

share|cite|improve this answer

I find the notation irritating, whereby the same letter, $p$, is used to refer to two or more different functions.

I could write an algebraic proof, but I wonder if a concrete example might not shed more light. Suppose we have $$ \begin{align} P(X=0\ \&\ Y=0) = 1/10 & & P(X=0\ \&\ Y=1) = 2/10 \\ \\ P(X=1\ \&\ Y=0) = 3/10 & & P(X=1\ \&\ Y=1) = 4/10 \end{align} $$ Then $P(X=0)=4/10$ and $P(X=1)= 6/10$. So the first sum above is $$ \underbrace{(1/10)\log (4/10) + (3/10)\log(4/10)} + \underbrace{(2/10)\log(6/10) + (4/10)\log(6/10) }. $$ The second sum above is $$ (4/10)\log(4/10) + (6/10)\log(6/10). $$ So it's really just the distributive law.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.