I was reviewing self information and then came to realize that there is one idea that I have that I believe should be wrong but don't know why.
Let self-information associated with a random variable realization be:
$$I_{X}(x) = \log{\frac{1}{P_{X}(x)}}$$
The main idea is that realizing rare events contains more information that realizing event that you knew would occur for sure, being because rare events tell us more information about the distribution than what we already knew about it.
However, for extremely low values of $P_{X}(x)$ I find that justification (or intuition) for defining $I_{X}(x)$ the way it was just weird or wrong.
For example, say that I was trying to learn a distribution by observing values coming from it every so often and applying some learning algorithm or sampling technique. Then, an event that is extremely rare happens and I update what I have learned so far about the distribution taking into account the rare event. However, since the event is so rare, we update what we have learned and if the learning procedure makes sense, I should that it should not change its estimate of the distribution too much, because otherwise it wouldn't be consistent that this event is actually rare. Therefore, that means for me that we could have just not seen the event and it would have been just as good off. We might never see that event again anyway. So event with low probability actually didn't tell us that much about that distribution itself.
This intuition is exactly the opposite to how the definition of information is justified. Does someone know why my argument does not make sense? Is it just for events that basically have a probability so close to zero that they are basically zero?