One form of Jensen's inequality for the finite case, tells us that

$$ \sum_{x \in X} p(x) \log q(x) \leq \log\sum_{x \in X} p(x) \cdot q(x) $$

For positive p(x), and $\sum_{x \in X} p(x) = 1$, $q(x)$ real, and $X$ finite. I am using the $\log$, but any concave function could be substituted.

or the probabilistic version:

$$ \mathbb{E}( \log X) \leq \log \mathbb{E}(X)$$ Where $\mathbb{E}$ is the expectation of $X$.

However, is this inequality true for countable $X$? The book I'm reading (elements of information theory, 2006), seems to prove it for the finite case, but uses the countable case without mentioning it.

Also on wikipedia the it seems the first inequality in my post is only for the finite case, whereas the probablistic verson makes no mention of cardinality of the probability space.

  • 1
    $\begingroup$ ${\rm E}[f(X)]\leq f({\rm E}[X])$ holds for any convex function $f$ and any random variable as long as $X$ is integrable, i.e. ${\rm E}[|X|]<\infty$. $\endgroup$ – Stefan Hansen Feb 13 '14 at 11:55
  • $\begingroup$ Thanks @Stefan, I suspected as much! Do you or anyone else have any literature about this? $\endgroup$ – Fenno Vermeij Feb 13 '14 at 12:10
  • 1
    $\begingroup$ I think Measures, Integrals and Martingales by Schilling has a proof of this. $\endgroup$ – Stefan Hansen Feb 13 '14 at 12:22
  • $\begingroup$ It's also Exercise 3.3.8 in Cohn's Measure Theory. $\endgroup$ – user21467 Feb 13 '14 at 16:51
  • $\begingroup$ You can look at this MSE answer: math.stackexchange.com/questions/513951/… $\endgroup$ – M Turgeon Feb 21 '14 at 14:48

Sketch, for the case where $X$ has support over $\mathbb{N}$:

Consider the variables $Y_n= \min(X,n)$ for $n=1,2 \cdots$

Then $Y_n$ has finite support, and so we have $\mathbb{E}( \log Y_n) \leq \log \mathbb{E} (Y_n)$

But $Y_n$ converges in distribution to $X$ as $n\to \infty$. Then $\log \mathbb{E}( Y_n) \to \log \mathbb{E}( X) $ and $\mathbb{E}( \log Y_n)\to \mathbb{E}( \log X) $. This implies $\mathbb{E}( \log X) \leq \log \mathbb{E} (X)$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.