Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Information theory is not at all my field of expertise, so maybe my question will be a bit naive.

As said in title, I would like to quantify the gain of information of a new information.

For instance, if I have a binary random event:



If I finally know that $X=0$ (situation 1), I do not gain much information. But if get to know that $X=1$ (situation 2), the situation changes a lot (I gain a lot of information?).

However if I compute the difference of entropy between the original situation and the final one. I get the same difference.

If I say that the fact of getting this new information is itself random event S (with probability $0.5$), s.t:

Situation 1:

$P(S=0 & X=0) = 0.9*0.5$

$P(S=0 & X=1) = 0.1*0.5$

$P(S=1 & X=0) = 0.5$

$P(S=1 & X=1)= 0$

Situation 2:

$P(S=0 & X=0) = 0.9*0.5$

$P(S=0 & X=1) = 0.1*0.5$

$P(S=1 & X=0) = 0$

$P(S=1 & X=1)= 0.5$

The mutual information between the random event S and X is bigger in situation 2, which confirms the intuition that the information gain is bigger in situation 2.

Still, it is a bit weird to me to say that S is a random event and to arbitrary set its probability at 0.5.... (But it is the best that I found so far).

Could you please tell me if there is a more standard way to deal with this situation, and quantify the gain of information?

share|cite|improve this question
up vote 2 down vote accepted

I think this question is answered by something even simpler than entropy: self information (also called surprisal).

Recall that the entropy of a discrete random variable $X$ with probability of $P(X=x_i)=p_i$ is given by $H(X)=\sum_i p_i\log_2(1/p_i)$. We can also think of $H(X)=E_p[\log_2(1/p_i)]$ and now the thing we are averaging is the self-information of the value $x_i$, given by $\log_2(1/p_i)=-\log_2(p_i)$.

So in our case the self information of 0 is $-\log_2(0.9)\approx 0.15$ bits and the self information of 1 is $-\log_2(0.1)\approx 3.3$ bits. So learning 1 instead of 0 gives us approximately 3.15 more bits of information.

share|cite|improve this answer
I see, this is very interesting. I think that answers my question. But, do you know if it is possible to generalize self information to a more complicated case. For instance, what if my new information is itself a probability. For instance, if someone rolls a dice, and tells me it is a 5 with 0.3 prob and a 6 with 0.7 prob... Maybe I should ask another question about it.... – oli Sep 27 '12 at 17:16
One thing to be careful is that information theory only works for probabilistic models, so set up your model precisely and then we should be able to tell what to use. For a dice here's one way to think about it: you could calculate the entropy of a standard die (think of it as how much information you gain on average upon seeing the result) and then the entropy of the new setup. The difference is in some sense the change in information you expect to learn. – Martin Leslie Sep 27 '12 at 17:34
If we have a model of the whole situation then you might also be interested in the conditional entropy (see the chain rule section of the wikipedia article for a good way to think about it). – Martin Leslie Sep 27 '12 at 17:35

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.