Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Consider the following problem:

What is $I(X;Y)$ where $X$ is the outcome of a roll of a fair 6-sided die and $Y$ is whether the outcome of THAT SAME ROLL was even or odd?

Intuitively, I thought that $I(X;Y) = H(X) + H(Y) - H(X, Y)$ - that is, when given even or odd, we cut $X$ in half (so we know the number was either 1, 3, 5 or 2, 4, 6). However, I'm slightly confused (as my professor argued that the correct answer was 1).

His idea was that if I'm given the number rolled, then all of the even/odd information is contained within, so $I(X;Y) = 1$.

Maybe I don't understand mutual information correctly, an explanation would be appreciated.

share|improve this question

1 Answer 1

Since there are six equiprobable outcomes for $X$ and two equiprobable outcomes for $Y$, if entropy is measured in bits, $$H(X)=(\log 6)/(\log 2),\qquad H(Y)=1.$$ Since $Y$ is a deterministic function of $X$, the joint random variable $(X,Y)$ has six equiprobable outcomes, just as $X$ does. So, $H(X,Y)=(\log 6)/(\log 2)$ also. Therefore, $$I(X;Y)=H(X)+H(Y)-H(X,Y)=\frac{\log 6}{\log 2} + 1 - \frac{\log 6}{\log 2}=1:$$ since all the information in $Y$ is already in $X$, $$H(X,Y)=H(X)\ \ \ \Rightarrow\ \ \ I(X;Y)=H(Y).$$ The idea that knowing $Y$ cuts the number of possibilities for $X$ in half is correct, but translated into entropy, it's about $H(X\mid Y)$: $$ H(X\mid Y)=H(X,Y)-H(Y)=\frac{\log 6}{\log 2}-1=\frac{\log 3}{\log 2}: $$ once you know $Y$, there are only three equiprobable outcomes for $X$.

share|improve this answer
    
This is correct, but it would be slightly more readable if the denominators $\log 2$ were omitted (in Information Theory $\log$ denotes commonly the logarithm in base 2) –  leonbloy Feb 27 '13 at 18:23

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.