# Entropy: average information vs uncertainty

Entropy seems to be one of those ideas that is easy to get wrong. I often see entropy roughly described as "average information" or the amount of "uncertainty" in a process. These seem like contradictory ideas to me.

If $H(X) = \sum_x p(x) \log_2 \frac{1}{ p(x)}$ is the entropy of the random varialbe $X$ with (discrete) probability distribution $p$, then I understand that the term $\log_2 \frac{1}{ p(x)}$ can be thought of as "information". Therefore, it makes sense to view $H(X)$ precisely as the expectation of the information, i.e. how much information you expect to learn, on average.

So how does uncertainty come into this? If $H(X)$ is large, implying that that average amount of information is "large", how can the uncertainty also be large? Wouldn't larger average information mean less uncertainty? How do we reconcile these two notions?

• You may be mixing different meanings of "entropy". In particular, in physics, the higher the entropy, the less information you have. Entropy always increases because processes destroy information. But in your interpretation, Entropy is the expected information, thus information would increase with entropy. Commented Jan 10, 2018 at 20:41
• I think at least some of the time, when people talk about entropy as uncertainty they're referring to how uncertain you are before seeing a sample. Commented Jan 10, 2018 at 21:07

Think of $$X$$ as the data, or observed value. Then, to say that before knowing $$X$$ we had much uncertainty is equivalent to say that after knowing $$X$$ we have gained much information.
By contrast, if we had -a priori- little uncertainty, then to know $$X$$ gives us -a posteriori- little information.