# Entropy and Expectation

Suppose that the Probability Mass Function of a random variable $$X$$ with values in $$A = \{1, 2, \dots\}$$ has nonincreasing probabilities, $$P(k + 1) \leq P(k)$$, for all $$k \geq 1$$. Show that, if $$H(X) < \infty$$, then $$\mathbb{E}[\log X] < \infty$$.

I can intuitively see that this is true since $$-\log$$ grows very quickly near the origin, much faster than linear. So, if $$H(X) = -\sum_{x=1}^\infty p(x)\log p(x)$$ is finite, then $$p(x)$$ must be shrinking quickly enough to modulate this and keep the sum finite. Hence, it decreases quickly enough to keep $$\mathbb{E}[\log X] = \sum_{x=1}^\infty p(x)\log x$$ finite as well. I can't seem to figure how to put this concept into rigorous math.

• so if $\mathbb E (\log X) = \sum_{x=1}^\infty p(x) \log p(x) = - H(X)$, isn't that trivial? Jan 27 at 16:20
• there is a difference between $\log(x)$ and $\log p(x)$... Jan 27 at 16:36
• okay, obviously. my bad. Jan 27 at 16:41

Since the sequence $$(p_k)$$ is decreasing, we have $$p_1 \geq \ldots \geq p_k$$ for all $$k.$$ If, for some $$k$$ we had that $$p_k > \dfrac{1}{k},$$ then we would reach that $$p_1 + \ldots + p_k > k \dfrac{1}{k} = 1,$$ which is impossible. Therefore, for all $$k$$ we have $$p_k \leq \dfrac{1}{k}.$$
The result now follows easily since then $$-p_k \log p_k \geq p_k \log k.$$ Q.E.D.
Since $$p(n)$$ is summable and montonically decreasing, we know that $$p(n)<\frac{1}{n}$$ for $$n$$ large, and this implies that $$-\log(p(n))>\log(n)$$. Hence, $$-log(p(n))p(n)\geq \log(n)p(n)$$ for $$n$$ large enough.
• You could have $p(n) > 1/n$ for infinitely many $n$ and the series $p(n)$ be summable. Also notice that you never used that $p(n)$ is decreasing... Jan 27 at 17:39