I gather for a series of four coin flips, if we get $H,H,H,H$, this has a probability of $\tfrac{1}{16}$, so we have information content $$\log_2 \frac{1}{\frac{1}{16}}$$

But for the rest of the events in the probability space, we have $$ \log_2\frac{1}{\frac{15}{16}}$$

which is much smaller. But why? Shouldn't a more probable event give us more information? I would think the occurrence of a more probable event tells us the system behaves as we expect and in this sense "informs" us.

My Question:

Why does the information content of a less probable event yield more bits than a more probable one?


It is quite the opposite. Imagine an event occurs with probability $1$. Then finding out that it has occured reveals nothing to you, you did not already know.

If on the other hand a very rare event occurs, receiving that information really tells you something unexpected thus revealing a lot of information you could not have deduced yourself.

  • $\begingroup$ Okay, suppose I flip a coin ten times in a row and get all heads. Extremely improbable, right? But suppose my coin is actually fair and that event was just a fluke. Then saying $\frac{1}{2^{10}}$ yields more information than $1-\frac{1}{2^{10}}$ .....that doesn't make sense. If I know the system is fair, it doesn't give me any information (in a qualitative sense) because that series of flips is not representative of the actual system. So the fluke 10 flips should give me less information (qualitative sense) because it doesn't characterize the system. For instance, the system would appear to $\endgroup$ – Stan Shunpike Apr 21 '15 at 18:04
  • $\begingroup$ have P(Heads) = 1. Note, I am sure you are correct because what you say matches the Wikipedia page and other sources I checked. I just don't see why my reasoning is wrong and I am trying to find out why it is. $\endgroup$ – Stan Shunpike Apr 21 '15 at 18:04
  • 3
    $\begingroup$ What you receive is not actually information about the probabilistic system. You are supposed to know that system perfectly and then measure how much information a specific event gives you about what has occured. To take an analogy: If you are to guess a concrete $3$-digit code, where the first digit is always $9$, the second is either $1,2,3$ or $4$, and the last is any of the ten digits. Each digit is equiprobable among its choices. Then if someone tells you, that the first digit is $9$, you learn $0$ bits. If someone tells you the second digit is $3$, you actually learn $2$ bits. $\endgroup$ – String Apr 21 '15 at 18:18
  • $\begingroup$ If someone tells you the last digit is $8$, you learn approximately $3.3$ bits theoretically. @StanShunpike $\endgroup$ – String Apr 21 '15 at 18:18
  • 1
    $\begingroup$ @StanShunpike: I am delighted to hear that! $\endgroup$ – String Apr 21 '15 at 20:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.