Finite entropy and finite moments

Question

This question is about when a discrete distribution has finite entropy because it has finite moments.

Let $X$ be a discrete random variable (which can be positive, negative, and/or zero unless stated otherwise), and let— $$H(X) = \sum_n -\mathbb{P}[X=n] \ln(\mathbb{P}[X=n]),$$ be the Shannon entropy of $X$.

It is known that:

• If $X$ is integer-valued and has a finite variance, then $H(X)$ is finite (Massey 1988).
• If $X$ is integer-valued, is 0 or greater, and has a finite mean, then $H(X)$ is finite (Rioul 2022).

It is also known that a discrete distribution can have infinite Shannon entropy even if it's positive-valued (one example is some members of the zeta Dirichlet distribution); see also Devroye and Gravel 2020.

However, I believe that this is so because the infinite-entropy zeta Dirichlet distributions (I think) have an infinite $z$th moment for any real $z>0$.

Moreover, I believe that the Cauchy distribution has a finite entropy even though its mean is infinite because it does have a finite $z$th moment for some $z$ in $(0, 1)$, in fact for every $z$ in $(0, 1)$.

However, several questions remain on the relationship between finite entropy and finite moments.

Questions

• If $X$ is supported on the whole set of integers and is such that $\mathbb{E}[X^2]$ is infinite but $\mathbb{E}[X^z]$ is finite for some $z$ in $[1, 2)$, then is $H(X)$ finite? If not, under what additional conditions is $H(X)$ finite?
• If $X$ is supported on the integers or some subset thereof, such that $\mathbb{E}[X]$ is infinite but $\mathbb{E}[X^z]$ is finite for every $z$ in $(0, 1)$, then is $H(X)$ finite? If not, under what additional conditions is $H(X)$ finite?
• If $X$ is supported on the integers or some subset thereof, such that $\mathbb{E}[X]$ is infinite but $\mathbb{E}[X^z]$ is finite for some $z$ in $(0, 1)$, then is $H(X)$ finite? If not, under what additional conditions is $H(X)$ finite?
• If $X$ is supported on the integers or some subset thereof, such that $\mathbb{E}[X^z]$ is infinite for every real $z > 0$, then is $H(X)$ infinite?

Motivation

My motivation is to characterize the discrete distributions with finite entropy, since only finite-entropy distributions can be sampled in finite time on average (Knuth and Yao 1976).

References

• O. Rioul, "Variations on a Theme by Massey," in IEEE Transactions on Information Theory, doi: 10.1109/TIT.2022.3141264.

• Massey, J.L., "On the entropy of integer-valued random variables", 1988.

• Devroye, L., Gravel, C., "Random variate generation using only finitely many unbiased, independently and identically distributed random bits", arXiv:1502.02539v6 [cs.IT], 2020.

• Knuth, Donald E. and Andrew Chi-Chih Yao. "The complexity of nonuniform random number generation", in Algorithms and Complexity: New Directions and Recent Results, 1976.

Answer 1

I assume that $X$ takes non-negative integral values.

---

First, suppose that $E X^a<\infty$ for some $a>0$. Then for all $n\ge 0$, it holds either that $$ p_n\log \frac{1}{p_n}\le n^a p_n $$ or $$ p_n\le e^{-n^a}. $$ But in the latter case, we can also bound, say, $\log(1/p_n)\le \sqrt{p_n}$. So both upper bounds are summable, and we get $H(X)<\infty$.

---

Reciprocally, if we know that $E X^a=\infty$ for all $a>0$, there is not much we can say about $H(X)$. Consider for example $p_n=1/m^2$ if $n=e^m$ for some $m\ge 0$ and $p_n=0$ otherwise. You can check that $p_n$ satisfies the all-infinite-moments condition (in fact, the main term of the sum does not even converge to $0$), but $$ \sum_{n\le e^m} p_n\log(1/p_n)=\sum_{k\le m} \frac{\log(k^2)}{k^2}. $$ So $H(X)<\infty$.

Answer 2

This answer aims to avoid the problems highlighted by Matija on Feb 6, 2023 with an answer from md5.

The entropy of $X$ can be written as $$\tag1 H(X)=-\sum_{n=1}^\infty p_n\log p_n $$ where $p_n=\mathbb P\{X=n\}\,.$ The expectation of $X$ is $$\tag2 \mathbb E[X]=\sum_{n=1}^\infty n\,p_n\,. $$ The natural numbers can be partitioned into two sets $A,B$ as follows: let $A\subset\mathbb N$ be the set of all $n\in\mathbb N$ who satisfy $$\tag{A} -\log p_n=\log\frac1{p_n}\le n\,. $$ Then $B=\mathbb N\setminus A$ is the set of all numbers satisfying $n<-\log p_n\,,$ or equivalently, $$\tag{B} p_n<e^{-n}\,. $$ The sum over $n$ in $A$ is finite when $\mathbb E[X]$ is finite but the sum over $B$ needs further examinations.

The function $(0,1]\ni x\mapsto-x\log x$ is concave and has a global maximum at $x=1/e\,.$ This function is strictly increasing in $(0,1/e)\,.$ Since $p_n$ must converge to zero all those probabilities must ultimately be in $(0,1/e)\,.$ That is:

• for almost all (up to finitely many) $n\in B$ we get from (B) that
  $$\tag3 -p_n\,\log p_n<n\,e^{-n}\, $$ holds. Lets denote the finite subset of $B$ for which (3) does not hold by $C$ and the possibly infinite subset for which (3) holds by $D\,.$

Now we get \begin{align} H(X)&=-\sum_{n\in A}p_n\,\log p_n-\sum_{n\in B}p_n\,\log p_n\\[2mm] &\le\underbrace{\sum_{n\in A}n\, p_n}_{\le\,\mathbb E[X]\,<\infty} \;\underbrace{-\sum_{n\in C}p_n\,\log p_n}_{<\infty\,\text{($C$ is finite)}} \;\underbrace{-\sum_{n\in D}p_n\,\log p_n}_{\stackrel{(3)}\le\,\sum\limits_{n\in \mathbb N}ne^{-n}}\,. \end{align} It remains to recall that the sum $\sum\limits_{n\in \mathbb N}ne^{-n}$ is finite: From $$\tag4 \sum_{n=1}^\infty e^{-xn}=\frac1{e^x-1} $$ we deduce $$\tag5 \sum_{n=1}^\infty n\,e^{-xn}=-\frac{d}{dx}\frac1{e^x-1}=\frac{e^x}{(e^x-1)^2} $$ which equals $\sum\limits_{n\in \mathbb N}ne^{-n}$ for $x=1$ where it is finite. $$\tag*{$\Box$} \quad $$