I've been conducting a small experiment to test a few of my interpretations about mutual information, and I'm running into some difficulties.

I've created some MATLAB code that basically makes two sets of $10,000$ random, normally-distributed values, centred around a mean of $0$. I called these sets $X$ and $Y$. I then "bin" each of these sets into two histograms containing ten bins. From here, I can calculate the entropy of each set, and thus their "self" information; $$H(X) = I(X;X)$$ $$H(Y) = I(Y;Y)$$ Now, the next thing I wanted to test was the information between $X$ and $Y$. To do this, I created a bivariate histogram, which gives me a $10 \times 10$ grid of joint counts, which I can then divide by $10,000$ to get the joint probabilities of $X$ and $Y$ occurring at the same time. I then used the method shown here; http://www.scholarpedia.org/article/Mutual_information to calculate the information value.

Now, my intuition was telling me that, because $X$ and $Y$ are roughly the same, then $I(X;Y) = I(Y;X) \approx H(X) \approx H(Y)$. Now, it's definitely true that $I(X;Y) = I(Y;X)$, and I have shown that $H(X) \approx H(Y)$, but the values I get for $I(X;Y)$ are around $0.005$, whereas my entropy values are generally around $2.4$.

I'm not 100% sure where this discrepancy is coming from. Any insight into why this might be happening would be very appreciated!!


Now, my intuition was telling me that, because X and Y are roughly the same, then $I(X;Y)=I(Y;X)≈H(X)≈H(Y)$

That's wrong. If $X$ and $Y$ are independent (as it seems here), then $$I(X;Y)=H(X)-H(X|Y) = H(X)-H(X)=0$$ (which is consistent with the intuitive notion of mutual information: if $Y$ is independent from $X$, then $Y$ gives me no information about $X$ - and viceversa). The value you've got ($0.005$) is very small relative to the marginal entropies - hence, it looks right.

BTW: the mutual information is symmetric, so $I(X;Y)=I(Y;X)$ always, it does not matter that they are identically distributed.

  • $\begingroup$ I had been thinking that $X$ and $Y$ were independent of each other, which would account for the issues here. Thanks for verifying that to some extent!! :) $\endgroup$ – Jack Apr 10 '15 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.