# Finite entropy and finite moments

This question is about when a discrete distribution has finite entropy because it has finite moments.

Let $$X$$ be a discrete random variable (which can be positive, negative, and/or zero unless stated otherwise), and let— $$H(X) = \sum_n -\mathbb{P}[X=n] \ln(\mathbb{P}[X=n]),$$ be the Shannon entropy of $$X$$.

It is known that:

• If $$X$$ is integer-valued and has a finite variance, then $$H(X)$$ is finite (Massey 1988).
• If $$X$$ is integer-valued, is 0 or greater, and has a finite mean, then $$H(X)$$ is finite (Rioul 2022).

It is also known that a discrete distribution can have infinite Shannon entropy even if it's positive-valued (one example is some members of the zeta Dirichlet distribution); see also Devroye and Gravel 2020.

However, I believe that this is so because the infinite-entropy zeta Dirichlet distributions (I think) have an infinite $$z$$th moment for any real $$z>0$$.

Moreover, I believe that the Cauchy distribution has a finite entropy even though its mean is infinite because it does have a finite $$z$$th moment for some $$z$$ in $$(0, 1)$$, in fact for every $$z$$ in $$(0, 1)$$.

However, several questions remain on the relationship between finite entropy and finite moments.

## Questions

• If $$X$$ is supported on the whole set of integers and is such that $$\mathbb{E}[X^2]$$ is infinite but $$\mathbb{E}[X^z]$$ is finite for some $$z$$ in $$[1, 2)$$, then is $$H(X)$$ finite? If not, under what additional conditions is $$H(X)$$ finite?
• If $$X$$ is supported on the integers or some subset thereof, such that $$\mathbb{E}[X]$$ is infinite but $$\mathbb{E}[X^z]$$ is finite for every $$z$$ in $$(0, 1)$$, then is $$H(X)$$ finite? If not, under what additional conditions is $$H(X)$$ finite?
• If $$X$$ is supported on the integers or some subset thereof, such that $$\mathbb{E}[X]$$ is infinite but $$\mathbb{E}[X^z]$$ is finite for some $$z$$ in $$(0, 1)$$, then is $$H(X)$$ finite? If not, under what additional conditions is $$H(X)$$ finite?
• If $$X$$ is supported on the integers or some subset thereof, such that $$\mathbb{E}[X^z]$$ is infinite for every real $$z > 0$$, then is $$H(X)$$ infinite?

## Motivation

My motivation is to characterize the discrete distributions with finite entropy, since only finite-entropy distributions can be sampled in finite time on average (Knuth and Yao 1976).

## References

• Thanks for the nice question! However, as explained here, the entropy of positive integer valued random variables can be infinite. Please edit the question to avoid confusion. Commented Feb 6, 2023 at 11:31
• @Matija: I don't see how this question should be edited, but I have tried. Commented Feb 6, 2023 at 15:40
• Can you please tell me what result exactly in Rioul 2022 you reference? The problem is that the assertion as you have written it is false. Commented Feb 6, 2023 at 15:42
• @Matija: It turned out to be theorem 8, which also assumes a finite mean. I have corrected this post. Commented Feb 6, 2023 at 16:19
• That answers the 1st question. Seems the answer to the 3rd question is worth a publication, since this would (and I guess, will) extend the results by Rioul. I have a construction in mind to negatively answer the 4th question. However, as pointed out before, we have to work with $\mathbb E[|X|^z]$. Recall that $\mathbb E[X]$ is well-defined iff $\mathbb E[\max(0,X)]<\infty$ or $\mathbb E[-\min(0,X)]<\infty$, so $|\mathbb E[x]|<\infty$ iff $\mathbb E[|X|]<\infty$, i.e. we don't loose anything by working with the absolute, and it's always defined. If 2 out of 4 are okay, I'll write it down. Commented Feb 6, 2023 at 16:45

I assume that $$X$$ takes non-negative integral values.

First, suppose that $$E X^a<\infty$$ for some $$a>0$$. Then for all $$n\ge 0$$, it holds either that $$p_n\log \frac{1}{p_n}\le n^a p_n$$ or $$p_n\le e^{-n^a}.$$ But in the latter case, we can also bound, say, $$\log(1/p_n)\le \sqrt{p_n}$$. So both upper bounds are summable, and we get $$H(X)<\infty$$.

Reciprocally, if we know that $$E X^a=\infty$$ for all $$a>0$$, there is not much we can say about $$H(X)$$. Consider for example $$p_n=1/m^2$$ if $$n=e^m$$ for some $$m\ge 0$$ and $$p_n=0$$ otherwise. You can check that $$p_n$$ satisfies the all-infinite-moments condition (in fact, the main term of the sum does not even converge to $$0$$), but $$\sum_{n\le e^m} p_n\log(1/p_n)=\sum_{k\le m} \frac{\log(k^2)}{k^2}.$$ So $$H(X)<\infty$$.

• Thank you for the response. Can you answer for the case when $X$ can be negative as well as positive? All along I intended the case when $X$ can be positive and/or negative, unless stated otherwise in the question, especially the case when $X$ can take any positive or negative integer. Commented Feb 18, 2022 at 18:27
• @PeterO. - so what does the moment condition correspond to now? Should we put absolute values?
– md5
Commented Feb 18, 2022 at 19:04
• Only $z$th moments for any real $z > 0$, and the absolute values of the moments are taken (e.g., where I said "$\mathbb{E}[X^z] = \infty$", I meant "$\mathbb{E}[X^z]$ is infinite" or "$\mathbb{E}[X^z]$ is positive infinity or negative infinity"). Commented Feb 18, 2022 at 19:12
• @PeterO. - but this is not even defined if the exponent is not an integer
– md5
Commented Feb 18, 2022 at 20:31
• @md5 If you use "either or", one of the inequalities should be strict. More importantly, I don't understand your $\log(1/p_n)\le\sqrt{p_n}$ bound. Say, $X$ is Bernoulli with $3/4$, so the choice of $a$ doesn't matter. For $n=0$ we have $p_0=1/4<1=e^{-0}$, but $\log(1/p_0)>1>1/2=\sqrt{p_0}$. Please correct the answer to avoid future confusion. Commented Feb 6, 2023 at 11:55

This answer aims to avoid the problems highlighted by Matija on Feb 6, 2023 with an answer from md5.

The entropy of $$X$$ can be written as $$\tag1 H(X)=-\sum_{n=1}^\infty p_n\log p_n$$ where $$p_n=\mathbb P\{X=n\}\,.$$ The expectation of $$X$$ is $$\tag2 \mathbb E[X]=\sum_{n=1}^\infty n\,p_n\,.$$ The natural numbers can be partitioned into two sets $$A,B$$ as follows: let $$A\subset\mathbb N$$ be the set of all $$n\in\mathbb N$$ who satisfy $$\tag{A} -\log p_n=\log\frac1{p_n}\le n\,.$$ Then $$B=\mathbb N\setminus A$$ is the set of all numbers satisfying $$n<-\log p_n\,,$$ or equivalently, $$\tag{B} p_n The sum over $$n$$ in $$A$$ is finite when $$\mathbb E[X]$$ is finite but the sum over $$B$$ needs further examinations.

The function $$(0,1]\ni x\mapsto-x\log x$$ is concave and has a global maximum at $$x=1/e\,.$$ This function is strictly increasing in $$(0,1/e)\,.$$ Since $$p_n$$ must converge to zero all those probabilities must ultimately be in $$(0,1/e)\,.$$ That is:

• for almost all (up to finitely many) $$n\in B$$ we get from (B) that
$$\tag3 -p_n\,\log p_n holds. Lets denote the finite subset of $$B$$ for which (3) does not hold by $$C$$ and the possibly infinite subset for which (3) holds by $$D\,.$$

Now we get \begin{align} H(X)&=-\sum_{n\in A}p_n\,\log p_n-\sum_{n\in B}p_n\,\log p_n\\[2mm] &\le\underbrace{\sum_{n\in A}n\, p_n}_{\le\,\mathbb E[X]\,<\infty} \;\underbrace{-\sum_{n\in C}p_n\,\log p_n}_{<\infty\,\text{(C is finite)}} \;\underbrace{-\sum_{n\in D}p_n\,\log p_n}_{\stackrel{(3)}\le\,\sum\limits_{n\in \mathbb N}ne^{-n}}\,. \end{align} It remains to recall that the sum $$\sum\limits_{n\in \mathbb N}ne^{-n}$$ is finite: From $$\tag4 \sum_{n=1}^\infty e^{-xn}=\frac1{e^x-1}$$ we deduce $$\tag5 \sum_{n=1}^\infty n\,e^{-xn}=-\frac{d}{dx}\frac1{e^x-1}=\frac{e^x}{(e^x-1)^2}$$ which equals $$\sum\limits_{n\in \mathbb N}ne^{-n}$$ for $$x=1$$ where it is finite. $$\tag*{\Box} \quad$$