Justifying $\log{\frac{1}{P_{X}(x)}}$ as the measure of self information

Question

I was reviewing self information and then came to realize that there is one idea that I have that I believe should be wrong but don't know why.

Let self-information associated with a random variable realization be:

$$I_{X}(x) = \log{\frac{1}{P_{X}(x)}}$$

The main idea is that realizing rare events contains more information that realizing event that you knew would occur for sure, being because rare events tell us more information about the distribution than what we already knew about it.

However, for extremely low values of $P_{X}(x)$ I find that justification (or intuition) for defining $I_{X}(x)$ the way it was just weird or wrong.

For example, say that I was trying to learn a distribution by observing values coming from it every so often and applying some learning algorithm or sampling technique. Then, an event that is extremely rare happens and I update what I have learned so far about the distribution taking into account the rare event. However, since the event is so rare, we update what we have learned and if the learning procedure makes sense, I should that it should not change its estimate of the distribution too much, because otherwise it wouldn't be consistent that this event is actually rare. Therefore, that means for me that we could have just not seen the event and it would have been just as good off. We might never see that event again anyway. So event with low probability actually didn't tell us that much about that distribution itself.

This intuition is exactly the opposite to how the definition of information is justified. Does someone know why my argument does not make sense? Is it just for events that basically have a probability so close to zero that they are basically zero?

Answer 1

There is a good example on page 71 of David MacKay's book Information Theory, Inference, and Learning Algorithms (link to legal download page). Imagine a game of battleships on a chessboard where there is only one ship and 63 empty squares. As we ask "is it on square x?" for each of the squares, the answer is probably "no", so a "no" answer gives us little information (the figures are in the book itself). If the answer is "yes" then we suddenly get a lot of information (game over). In any case the sum of all the self-informations is always six bits.

If you could ask questions like "Is it in the top half?" then the answer to each question would have information content of one bit. In this case we would get to the answer more quickly than by asking about individual squares, but the sum of all the self-informations is still exactly six bits.