Entropy: average information vs uncertainty

Entropy seems to be one of those ideas that is easy to get wrong. I often see entropy roughly described as "average information" or the amount of "uncertainty" in a process. These seem like contradictory ideas to me.

If $H(X) = \sum_x p(x) \log_2 \frac{1}{ p(x)}$ is the entropy of the random varialbe $X$ with (discrete) probability distribution $p$, then I understand that the term $\log_2 \frac{1}{ p(x)}$ can be thought of as "information". Therefore, it makes sense to view $H(X)$ precisely as the expectation of the information, i.e. how much information you expect to learn, on average.

So how does uncertainty come into this? If $H(X)$ is large, implying that that average amount of information is "large", how can the uncertainty also be large? Wouldn't larger average information mean less uncertainty? How do we reconcile these two notions?

• You may be mixing different meanings of "entropy". In particular, in physics, the higher the entropy, the less information you have. Entropy always increases because processes destroy information. But in your interpretation, Entropy is the expected information, thus information would increase with entropy. – Paul Sinclair Jan 10 '18 at 20:41
• I think at least some of the time, when people talk about entropy as uncertainty they're referring to how uncertain you are before seeing a sample. – Qiaochu Yuan Jan 10 '18 at 21:07

Think of $X$ as the data, or observed value. Then to say that after knowing $X$ we gain much information is equivalent to say that before knowing $X$ we had much uncertainty.
By contrast, if we had -a priori- little uncertainty, then to know $X$ gives us -a posteriori- little information.