Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm a bit lost on understanding this formula in my bioinformatics text, and I appreciate any tips or advice.

Mutual Information, $\operatorname{MI}(X; Y)$ is: $$ \mu = \sum_x \sum_y p(xy) \log \left( \frac{p(xy)}{p(x) p(y)} \right). $$ If $X$ and $Y$ are independent r.v.'s, then $\operatorname{MI} = 0$.

Original link.

By the way, It's part of this larger picture

Thank You very much.

share|cite|improve this question
Many thanks for the edit, I will learn this notation. – Adel Sep 19 '11 at 3:44
@all, I don't understand the downvote. It seems like a reasonable question. – Srivatsan Sep 19 '11 at 3:46
up vote 3 down vote accepted

Well, without a little more background on which parts you understand and which parts you don't, it's hard to gauge the level at which to pitch the explanation. However, the situation is this:

We have two discrete random variables X and Y, which in this case represent the nucleotide at a a particular position in a DNA sequence, so $X$ and $Y$ take on the values A, C, G, and T.

In the equation above, $p(xy)$ denotes ${\rm Pr}[X = x,\; Y = y]$, i.e. the probability that X takes on the value x and Y takes on the value Y. $p(x)$ denotes ${\rm Pr}[X = x]$, the probability that $X$ takes on the value $x$ without regard to $Y$, and similarly for $p(y)$. So, in other words, we could write this in more readable form as

$${\rm MI}(X; Y) = \sum_{x \in \{A, C, G, T\}} \sum_{y \in \{A, C, G, T\}} {\rm Pr}[X = x,\; Y = y] \log \left( \frac{{\rm Pr}[X = x, Y = y]}{{\rm Pr}[X = x] {\rm Pr}[Y = y]}\right)$$

If $X$ and $Y$ are independent, then

$${\rm Pr}[X = x,\; Y = y] = {\rm Pr}[X = x] {\rm Pr}[Y = y]$$

so we have

$$\log \left( \frac{{\rm Pr}[X = x, Y = y]}{{\rm Pr}[X = x] {\rm Pr}[Y = y]}\right) = \log 1 = 0$$

for every possible choice of $x$ and $y$, and therefore when we calculate $MI(X; Y)$ we're just adding up sixteen zeros, which explains why it says that $MI(X; Y) = 0$ when $X$ and $Y$ are independent.

If you're still getting used to probabilities and statistics, you're not going to pick something like this up on the first try. Try drawing out a 4x4 grid with the various possibilities, filling it in with different probabilities, and calculating the mutual information to see how it works.

EDIT: You can try it out at

Just make sure only to edit the yellow cells. You can also check out the Wikipedia article:

share|cite|improve this answer
Thank You very much - excellent answer and I am truly indebted. The re-written formula cleared it up for me(A,C,T,G). I'm much closer now and will read the wiki link too. And the spreadsheet is very helpful also. Thank You - you are awesome! – Adel Sep 19 '11 at 4:52
No problem -- just take this and go cure cancer and we'll be square. – Daniel McLaury Sep 19 '11 at 5:00
@user3296 Oh, don't worry. We're working on it. – MrGomez Oct 14 '11 at 0:15

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.