Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

The Shannon Entropy for an observation is given by $ -x \log_2(x)$. Why is the maximum entropy achieved at $x = \frac{1}{e}$, and not at $x = 0$? Could someone provide a logical explanation that justifies the mathematics?

share|cite|improve this question
There is no entropy for "an observation". And I wonder how you got that it's maximized at $x=1/e$ (?) – leonbloy Apr 3 '14 at 15:52
It is true that the maximum of $-xlog_2(x)$ is at $x=1/e$, as you can see by differentiating/graphing. But I don't see any relevance of that to entropy. – Martin Leslie Apr 3 '14 at 16:19
up vote 1 down vote accepted

Shannon Entropy, given a discrete probability distribution with probabilities $p_1,p_2,\dots,p_n$ is $$-\sum_i p_i\log_2(p_i).$$

This is not equal to the quantity you gave. If $n=1$ then we must have $p_1=1$ and then the Shannon Entropy is 0.

A relevant example is to figure out what $p_1$ maximizes Shannon entropy when $n=2$. This can be thought of as what coin (not necessarily fair) gives you the most information when you flip it.

share|cite|improve this answer

As Martin Leslie mentions, Shannon Entropy, given a discrete probability distribution with probabilities $p_1,p_2,\dots,p_n$ is $$H = -\sum_i p_i\log_2(p_i).$$

The term $-\log_2(p_i)$ is the length of the optimal encoding of variable $i$. So a string of $n$ variables will have an encoded length of $nH$, and $-sp_i\log_2(p_i)$ is the number of bits used in encoding variable $i$. The fact that $-x\log(x)$ is maximized at $1/e$ means that more bits will be used encoding variables of probability $1/e$ than any other probability. That's fairly interesting.

As to why the mininum is not at 0: $-x\log x$ is, again, proportional to the total number of bits used toward encoding a variable of probability $x$. While the encoding length of a variable of probability very close to 0 is very large, the number of times it will be used is very small, so it's at least not CLEAR that 0 should be the winner. And indeed, when you do the math it's not.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.