5
$\begingroup$

Entropy seems to be one of those ideas that is easy to get wrong. I often see entropy roughly described as "average information" or the amount of "uncertainty" in a process. These seem like contradictory ideas to me.

If $H(X) = \sum_x p(x) \log_2 \frac{1}{ p(x)}$ is the entropy of the random varialbe $X$ with (discrete) probability distribution $p$, then I understand that the term $\log_2 \frac{1}{ p(x)}$ can be thought of as "information". Therefore, it makes sense to view $H(X)$ precisely as the expectation of the information, i.e. how much information you expect to learn, on average.

So how does uncertainty come into this? If $H(X)$ is large, implying that that average amount of information is "large", how can the uncertainty also be large? Wouldn't larger average information mean less uncertainty? How do we reconcile these two notions?

$\endgroup$
2
  • $\begingroup$ You may be mixing different meanings of "entropy". In particular, in physics, the higher the entropy, the less information you have. Entropy always increases because processes destroy information. But in your interpretation, Entropy is the expected information, thus information would increase with entropy. $\endgroup$
    – Paul Sinclair
    Commented Jan 10, 2018 at 20:41
  • 2
    $\begingroup$ I think at least some of the time, when people talk about entropy as uncertainty they're referring to how uncertain you are before seeing a sample. $\endgroup$
    – Qiaochu Yuan
    Commented Jan 10, 2018 at 21:07

1 Answer 1

Reset to default
7
$\begingroup$

True, high entropy means high information, or high uncertainty.

How do we reconcile these two notions?

Think of $X$ as the data, or observed value. Then, to say that before knowing $X$ we had much uncertainty is equivalent to say that after knowing $X$ we have gained much information.

By contrast, if we had -a priori- little uncertainty, then to know $X$ gives us -a posteriori- little information.

Another important example is the coin (sequence of independent and equiprobable 0-1s). This "white noise" process is totally unpredictable, it has maximal uncertainty. Equivalently, each coin result gives us a lot information (1 bit). By contrast, if the coin were not fair, or if the trials had some dependence, then we could predict to some degree each coin result, then we'd had less uncertainty - and, equivalently, each coin result would give us less information (because part of the result could be predicted).

Notice that, in this interpretation, we are speaking of averages (so "little information" = "little average information" = "little entropy").

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .