Entropy: average information vs uncertainty Entropy seems to be one of those ideas that is easy to get wrong. I often see entropy roughly described as "average information" or the amount of "uncertainty" in a process. These seem like contradictory ideas to me.
If $H(X) = \sum_x p(x) \log_2 \frac{1}{ p(x)}$ is the entropy of the random varialbe $X$ with (discrete) probability distribution $p$, then I understand that the term $\log_2 \frac{1}{ p(x)}$ can be thought of as "information". Therefore, it makes sense to view $H(X)$ precisely as the expectation of the information, i.e. how much information you expect to learn, on average.
So how does uncertainty come into this? If $H(X)$ is large, implying that that average amount of information is "large", how can the uncertainty also be large? Wouldn't larger average information mean less uncertainty? How do we reconcile these two notions?
 A: True, high entropy means high information, or high uncertainty. 

How do we reconcile these two notions?

Think of $X$ as the data, or observed value. Then, to say 
that before knowing $X$ we had much uncertainty
is equivalent to say
that after knowing $X$ we have gained much information.
By contrast, if we had -a priori- little uncertainty, then to know $X$ gives us -a posteriori- little information.
Another important example is the coin (sequence of independent and equiprobable 0-1s). This "white noise" process is totally unpredictable, it has maximal uncertainty. Equivalently, each coin result gives us a lot information (1 bit). By contrast, if the coin were not fair, or if the trials had some dependence, then we could predict to some degree each coin result, then we'd had less uncertainty - and, equivalently, each coin ersult would give as less information (because part of the result could be predicted).
Notice that, in this interpretation, we are speaking of averages (so "little information" = "little average information" = "little entropy").
