# calculate entropy of joint distribution

beforehand,i want to congrat coming new year guys,wish all you everything best in your life,now i have a little problem and please help me,i know definition of entropy which has a formula

$$\sum\limits_{i=1}^{n} -p(m_{i})\log_{2}(p(m_{i}))$$

now suppose we have following table for joint distribution we know that marginal distribution of X is (1/2,1/4,1/8,1/8) and for y (1/4,1/4,1/4,1/4) i know how is calculating marginal distribution,we can calculate that Entropy $H(x)=7/4$ bits and $H(y)=2$ bits; but i have a question with following formula,there is given that

$$H(X\mid Y)= \sum_{i=1}^4 P(Y=i) H(X\mid Y=i)$$

and here is calculation process finally $H(X\mid Y)$ according to above formula

$$H(X\mid Y)=\frac{1}{4}\left( H(1/2,1/4,1/8,1/8)+H(1/4,1/2,1/8,1/8)+H(1/4,1/4,1/4,1/4)+H(1,0,0,0)\right)$$

so my question is how we have got last one?please help me

## 2 Answers

I think your question is where $$H(X|Y)=(1/4)H(1/2,1/4,1/8,1/8)+(1/4)H(1/4,1/2,1/8,1/8)$$ $$+(1/4)H(1/4,1/4,1/4,1/4)+(1/4)H(1,0,0,0)$$ comes from given $$H(X|Y)=\sum_i P(Y=i)H(X|Y=i).$$ First note that $P(Y=i)$ is $(1/4)$ for every $i$ (just sum across the row $Y=i$ in your picture above); this is where the $1/4$ in front of each $H(X|Y=i)$ comes from. Also note that $X|Y=i$ is a probability distribution, with $P(X=j|Y=i)=P(X=j,Y=i)/P(Y=i)$. So, for instance, $X|(Y=1)$ is given by $4(1/8,1/16,1/32,1/32)=(1/2,1/4,1/8,1/8)$ and $H(X|Y=1)$ is $H(1/2,1/4,1/8,1/8)$. Hence the first term in the sum for $H(X|Y)$ is what is written above. I hope this helps, as I'm not sure exactly what you were asking.

• yes exactly it is what i wanted thanks very much – dato datuashvili Dec 30 '11 at 14:45
• happy new year @yoyo – dato datuashvili Dec 30 '11 at 14:54

As per this formula H(X|Y)=∑P(Y=i)H(X|Y=i), we can calculate this expression. Lets take one example lets take case of p( Y = 2)H( X |Y=2) ; for this first part p( Y = 2) is pretty clear which is 1/4 as you already found. In second party we need to calculate joint distribution P( x | y = 2 ) ; this will be p ( x=1 | y=2) , p(x=2 | y=2), p( x=3 | y=2) , and p(x=4 | y=2) we need to find this expressions and we are done.

Remember P(x | y) = p(x,y)/p(y) given that x,y have some correlation. Hope this helps.