Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

The Shannon Entropy for an observation is given by $ -x \log_2(x)$. Why is the maximum entropy achieved at $x = \frac{1}{e}$, and not at $x = 0$? Could someone provide a logical explanation that justifies the mathematics?

share|improve this question
    
There is no entropy for "an observation". And I wonder how you got that it's maximized at $x=1/e$ (?) –  leonbloy Apr 3 at 15:52
    
It is true that the maximum of $-xlog_2(x)$ is at $x=1/e$, as you can see by differentiating/graphing. But I don't see any relevance of that to entropy. –  Martin Leslie Apr 3 at 16:19

1 Answer 1

up vote 1 down vote accepted

Shannon Entropy, given a discrete probability distribution with probabilities $p_1,p_2,\dots,p_n$ is $$-\sum_i p_i\log_2(p_i).$$

This is not equal to the quantity you gave. If $n=1$ then we must have $p_1=1$ and then the Shannon Entropy is 0.

A relevant example is to figure out what $p_1$ maximizes Shannon entropy when $n=2$. This can be thought of as what coin (not necessarily fair) gives you the most information when you flip it.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.