Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

beforehand,i want to congrat coming new year guys,wish all you everything best in your life,now i have a little problem and please help me,i know definition of entropy which has a formula

$$\sum\limits_{i=1}^{n} -p(m_{i})\log_{2}(p(m_{i}))$$

now suppose we have following table for joint distribution enter image description here

we know that marginal distribution of X is (1/2,1/4,1/8,1/8) and for y (1/4,1/4,1/4,1/4) i know how is calculating marginal distribution,we can calculate that Entropy $H(x)=7/4$ bits and $H(y)=2$ bits; but i have a question with following formula,there is given that

$$H(X\mid Y)= \sum_{i=1}^4 P(Y=i) H(X\mid Y=i)$$

and here is calculation process finally $H(X\mid Y)$ according to above formula

$$H(X\mid Y)=\frac{1}{4}\left( H(1/2,1/4,1/8,1/8)+H(1/4,1/2,1/8,1/8)+H(1/4,1/4,1/4,1/4)+H(1,0,0,0)\right)$$

so my question is how we have got last one?please help me

share|cite|improve this question
up vote 3 down vote accepted

I think your question is where $$ H(X|Y)=(1/4)H(1/2,1/4,1/8,1/8)+(1/4)H(1/4,1/2,1/8,1/8) $$ $$+(1/4)H(1/4,1/4,1/4,1/4)+(1/4)H(1,0,0,0)$$ comes from given $$ H(X|Y)=\sum_i P(Y=i)H(X|Y=i). $$ First note that $P(Y=i)$ is $(1/4)$ for every $i$ (just sum across the row $Y=i$ in your picture above); this is where the $1/4$ in front of each $H(X|Y=i)$ comes from. Also note that $X|Y=i$ is a probability distribution, with $P(X=j|Y=i)=P(X=j,Y=i)/P(Y=i)$. So, for instance, $X|(Y=1)$ is given by $4(1/8,1/16,1/32,1/32)=(1/2,1/4,1/8,1/8)$ and $H(X|Y=1)$ is $H(1/2,1/4,1/8,1/8)$. Hence the first term in the sum for $H(X|Y)$ is what is written above. I hope this helps, as I'm not sure exactly what you were asking.

share|cite|improve this answer
yes exactly it is what i wanted thanks very much – dato datuashvili Dec 30 '11 at 14:45
happy new year @yoyo – dato datuashvili Dec 30 '11 at 14:54

As per this formula H(X|Y)=∑P(Y=i)H(X|Y=i), we can calculate this expression. Lets take one example lets take case of p( Y = 2)H( X |Y=2) ; for this first part p( Y = 2) is pretty clear which is 1/4 as you already found. In second party we need to calculate joint distribution P( x | y = 2 ) ; this will be p ( x=1 | y=2) , p(x=2 | y=2), p( x=3 | y=2) , and p(x=4 | y=2) we need to find this expressions and we are done.

Remember P(x | y) = p(x,y)/p(y) given that x,y have some correlation. Hope this helps.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.