Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

For an RV $X$ with values on $\{1,2,\ldots\}$, I need to prove that the entropy is less than the EV: $H(X)\leq E(X)$ . I tried to bound the log but I'm not quite there. Appreciate any hint...

Thanks

share|improve this question
    
Managed to solve it, using $D(P||Q)\geq 0$ with the distribution $q_i = 2^{-i}$ . –  yoki Apr 10 '12 at 16:19
    
You can answer your own question. This would be interesting for user which will encounter this problem or a similar one, and the question won't be classified as unanswered anymore. –  Davide Giraudo Apr 10 '12 at 18:34

1 Answer 1

up vote 2 down vote accepted

As requested, I'm answering my own question:

It is known that the Divergence holds $D(P||Q)\geq 0$ , ie $\sum p_i \log \frac{p_i}{q_i}\geq 0$. so:

$\sum p_i \log p_i \geq \sum p_i \log q_i$

$-\sum p_i \log p_i \leq \sum p_i \log q_i^{-1}$

Choosing a probability distribution q: $q_i = 2^{-i}$ yields:

$-\sum p_i \log p_i \leq \sum p_i \log 2^i=\sum p_i \cdot i=EX$. Therefore: $H(X)\leq EX$

PS, $q$ is a distribution since $0\leq q_i\leq 1$ and $\sum q_i=1$

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.