I've been conducting a small experiment to test a few of my interpretations about mutual information, and I'm running into some difficulties.
I've created some MATLAB code that basically makes two sets of $10,000$ random, normally-distributed values, centred around a mean of $0$. I called these sets $X$ and $Y$. I then "bin" each of these sets into two histograms containing ten bins. From here, I can calculate the entropy of each set, and thus their "self" information; $$H(X) = I(X;X)$$ $$H(Y) = I(Y;Y)$$ Now, the next thing I wanted to test was the information between $X$ and $Y$. To do this, I created a bivariate histogram, which gives me a $10 \times 10$ grid of joint counts, which I can then divide by $10,000$ to get the joint probabilities of $X$ and $Y$ occurring at the same time. I then used the method shown here; http://www.scholarpedia.org/article/Mutual_information to calculate the information value.
Now, my intuition was telling me that, because $X$ and $Y$ are roughly the same, then $I(X;Y) = I(Y;X) \approx H(X) \approx H(Y)$. Now, it's definitely true that $I(X;Y) = I(Y;X)$, and I have shown that $H(X) \approx H(Y)$, but the values I get for $I(X;Y)$ are around $0.005$, whereas my entropy values are generally around $2.4$.
I'm not 100% sure where this discrepancy is coming from. Any insight into why this might be happening would be very appreciated!!