Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

For an RV $X$ with values on $\{1,2,\ldots\}$, I need to prove that the entropy is less than the EV: $H(X)\leq E(X)$ . I tried to bound the log but I'm not quite there. Appreciate any hint...


share|cite|improve this question
Managed to solve it, using $D(P||Q)\geq 0$ with the distribution $q_i = 2^{-i}$ . – yoki Apr 10 '12 at 16:19
You can answer your own question. This would be interesting for user which will encounter this problem or a similar one, and the question won't be classified as unanswered anymore. – Davide Giraudo Apr 10 '12 at 18:34
up vote 2 down vote accepted

As requested, I'm answering my own question:

It is known that the Divergence holds $D(P||Q)\geq 0$ , ie $\sum p_i \log \frac{p_i}{q_i}\geq 0$. so:

$\sum p_i \log p_i \geq \sum p_i \log q_i$

$-\sum p_i \log p_i \leq \sum p_i \log q_i^{-1}$

Choosing a probability distribution q: $q_i = 2^{-i}$ yields:

$-\sum p_i \log p_i \leq \sum p_i \log 2^i=\sum p_i \cdot i=EX$. Therefore: $H(X)\leq EX$

PS, $q$ is a distribution since $0\leq q_i\leq 1$ and $\sum q_i=1$

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.