Entropy seems to be one of those ideas that is easy to get wrong. I often see entropy roughly described as "average information" or the amount of "uncertainty" in a process. These seem like contradictory ideas to me.

If $H(X) = \sum_x p(x) \log_2 \frac{1}{ p(x)}$ is the entropy of the random varialbe $X$ with (discrete) probability distribution $p$, then I understand that the term $\log_2 \frac{1}{ p(x)}$ can be thought of as "information". Therefore, it makes sense to view $H(X)$ precisely as the expectation of the information, i.e. how much information you expect to learn, on average.

So how does uncertainty come into this? If $H(X)$ is large, implying that that average amount of information is "large", how can the uncertainty also be large? Wouldn't larger average information mean less uncertainty? How do we reconcile these two notions?

  • $\begingroup$ You may be mixing different meanings of "entropy". In particular, in physics, the higher the entropy, the less information you have. Entropy always increases because processes destroy information. But in your interpretation, Entropy is the expected information, thus information would increase with entropy. $\endgroup$ – Paul Sinclair Jan 10 '18 at 20:41
  • 1
    $\begingroup$ I think at least some of the time, when people talk about entropy as uncertainty they're referring to how uncertain you are before seeing a sample. $\endgroup$ – Qiaochu Yuan Jan 10 '18 at 21:07

High entropy means high information, or high uncertainty.

How do we reconcile these two notions?

Think of $X$ as the data, or observed value. Then to say that after knowing $X$ we gain much information is equivalent to say that before knowing $X$ we had much uncertainty.

By contrast, if we had -a priori- little uncertainty, then to know $X$ gives us -a posteriori- little information.

Another important example is the coin (sequence of independent and equiprobable 0-1s). This "white noise" process is totally unpredictable, it has maximal uncertainty. Equivalently, each coin result gives us a lot information (1 bit). By contrast, if the coin were not fair, or if the trials had some dependence, then we could predict to some degree each coin result, then we'd had less uncertainty - and, equivalently, each coin ersult would give as less information (because part of the result could be predicted).

Notice that, in this interpretation, we are speaking of averages (so "little information" = "little average information" = "little entropy").


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.