0
$\begingroup$

beforehand,i want to congrat coming new year guys,wish all you everything best in your life,now i have a little problem and please help me,i know definition of entropy which has a formula

$$\sum\limits_{i=1}^{n} -p(m_{i})\log_{2}(p(m_{i}))$$

now suppose we have following table for joint distribution enter image description here

we know that marginal distribution of X is (1/2,1/4,1/8,1/8) and for y (1/4,1/4,1/4,1/4) i know how is calculating marginal distribution,we can calculate that Entropy $H(x)=7/4$ bits and $H(y)=2$ bits; but i have a question with following formula,there is given that

$$H(X\mid Y)= \sum_{i=1}^4 P(Y=i) H(X\mid Y=i)$$

and here is calculation process finally $H(X\mid Y)$ according to above formula

$$H(X\mid Y)=\frac{1}{4}\left( H(1/2,1/4,1/8,1/8)+H(1/4,1/2,1/8,1/8)+H(1/4,1/4,1/4,1/4)+H(1,0,0,0)\right)$$

so my question is how we have got last one?please help me

$\endgroup$
3
$\begingroup$

I think your question is where $$ H(X|Y)=(1/4)H(1/2,1/4,1/8,1/8)+(1/4)H(1/4,1/2,1/8,1/8) $$ $$+(1/4)H(1/4,1/4,1/4,1/4)+(1/4)H(1,0,0,0)$$ comes from given $$ H(X|Y)=\sum_i P(Y=i)H(X|Y=i). $$ First note that $P(Y=i)$ is $(1/4)$ for every $i$ (just sum across the row $Y=i$ in your picture above); this is where the $1/4$ in front of each $H(X|Y=i)$ comes from. Also note that $X|Y=i$ is a probability distribution, with $P(X=j|Y=i)=P(X=j,Y=i)/P(Y=i)$. So, for instance, $X|(Y=1)$ is given by $4(1/8,1/16,1/32,1/32)=(1/2,1/4,1/8,1/8)$ and $H(X|Y=1)$ is $H(1/2,1/4,1/8,1/8)$. Hence the first term in the sum for $H(X|Y)$ is what is written above. I hope this helps, as I'm not sure exactly what you were asking.

$\endgroup$
  • $\begingroup$ yes exactly it is what i wanted thanks very much $\endgroup$ – dato datuashvili Dec 30 '11 at 14:45
  • $\begingroup$ happy new year @yoyo $\endgroup$ – dato datuashvili Dec 30 '11 at 14:54
0
$\begingroup$

As per this formula H(X|Y)=∑P(Y=i)H(X|Y=i), we can calculate this expression. Lets take one example lets take case of p( Y = 2)H( X |Y=2) ; for this first part p( Y = 2) is pretty clear which is 1/4 as you already found. In second party we need to calculate joint distribution P( x | y = 2 ) ; this will be p ( x=1 | y=2) , p(x=2 | y=2), p( x=3 | y=2) , and p(x=4 | y=2) we need to find this expressions and we are done.

Remember P(x | y) = p(x,y)/p(y) given that x,y have some correlation. Hope this helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.