Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Consider the following problem:

What is $I(X;Y)$ where $X$ is the outcome of a roll of a fair 6-sided die and $Y$ is whether the outcome of THAT SAME ROLL was even or odd?

Intuitively, I thought that $I(X;Y) = H(X) + H(Y) - H(X, Y)$ - that is, when given even or odd, we cut $X$ in half (so we know the number was either 1, 3, 5 or 2, 4, 6). However, I'm slightly confused (as my professor argued that the correct answer was 1).

His idea was that if I'm given the number rolled, then all of the even/odd information is contained within, so $I(X;Y) = 1$.

Maybe I don't understand mutual information correctly, an explanation would be appreciated.

share|cite|improve this question

Since there are six equiprobable outcomes for $X$ and two equiprobable outcomes for $Y$, if entropy is measured in bits, $$H(X)=(\log 6)/(\log 2),\qquad H(Y)=1.$$ Since $Y$ is a deterministic function of $X$, the joint random variable $(X,Y)$ has six equiprobable outcomes, just as $X$ does. So, $H(X,Y)=(\log 6)/(\log 2)$ also. Therefore, $$I(X;Y)=H(X)+H(Y)-H(X,Y)=\frac{\log 6}{\log 2} + 1 - \frac{\log 6}{\log 2}=1:$$ since all the information in $Y$ is already in $X$, $$H(X,Y)=H(X)\ \ \ \Rightarrow\ \ \ I(X;Y)=H(Y).$$ The idea that knowing $Y$ cuts the number of possibilities for $X$ in half is correct, but translated into entropy, it's about $H(X\mid Y)$: $$ H(X\mid Y)=H(X,Y)-H(Y)=\frac{\log 6}{\log 2}-1=\frac{\log 3}{\log 2}: $$ once you know $Y$, there are only three equiprobable outcomes for $X$.

share|cite|improve this answer
This is correct, but it would be slightly more readable if the denominators $\log 2$ were omitted (in Information Theory $\log$ denotes commonly the logarithm in base 2) – leonbloy Feb 27 '13 at 18:23

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.