Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I know that the maximum possible Shannon Entropy for an alphabet $X$ is $\log|X|$, where Shannon Entropy is:

$$H(X) = - \sum_{x \in X} \; p(x) \log p(x)$$

but how is this upper limit computed?

share|cite|improve this question
No, its $\log |X|$ if $p(x)>0$ for all $x\in X$; otherwise its $|\{x:p(x)>0\}|$. Write $H(X) = \sum_{x \in X} \; p(x) \log(1/ p(x))$, and then use Jensen's inequality. – Ashok Jul 9 '12 at 11:06
Good point, stupid fingers - will update the question to fix the $log|X|$ issue. I was trying to differentiate and set to zero, I think I see how this approach works, thank you – Brabster Jul 9 '12 at 11:17

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.