5
$\begingroup$

This question is about when a discrete distribution has finite entropy because it has finite moments.

Let $X$ be a discrete random variable (which can be positive, negative, and/or zero unless stated otherwise), and let— $$H(X) = \sum_n -\mathbb{P}[X=n] \ln(\mathbb{P}[X=n]),$$ be the Shannon entropy of $X$.

It is known that:

  • If $X$ is integer-valued and has a finite variance, then $H(X)$ is finite (Massey 1988).
  • If $X$ is integer-valued and 0 or greater, then $H(X)$ is finite (Rioul 2022).

It is also known that not all discrete distributions have a finite Shannon entropy (one example is some members of the zeta Dirichlet distribution); see also Devroye and Gravel 2020.

However, I believe that this is so because the infinite-entropy zeta Dirichlet distributions (I think) have an infinite $z$th moment for any real $z>0$.

Moreover, I believe that the Cauchy distribution has a finite entropy even though its mean is infinite because it does have a finite $z$th moment for some $z$ in $(0, 1)$, in fact for every $z$ in $(0, 1)$.

However, several questions remain on the relationship between finite entropy and finite moments.

Questions

  • If $X$ is supported on the whole set of integers and is such that $\mathbb{E}[X^2]$ is infinite but $\mathbb{E}[X^z]$ is finite for some $z$ in $[1, 2)$, then is $H(X)$ finite? If not, under what additional conditions is $H(X)$ finite?
  • If $X$ is supported on the integers or some subset thereof, such that $\mathbb{E}[X]$ is infinite but $\mathbb{E}[X^z]$ is finite for every $z$ in $(0, 1)$, then is $H(X)$ finite? If not, under what additional conditions is $H(X)$ finite?
  • If $X$ is supported on the integers or some subset thereof, such that $\mathbb{E}[X]$ is infinite but $\mathbb{E}[X^z]$ is finite for some $z$ in $(0, 1)$, then is $H(X)$ finite? If not, under what additional conditions is $H(X)$ finite?
  • If $X$ is supported on the integers or some subset thereof, such that $\mathbb{E}[X^z]$ is infinite for every real $z > 0$, then is $H(X)$ infinite?

Motivation

My motivation is to characterize the discrete distributions with finite entropy, since only finite-entropy distributions can be sampled in finite time on average (Knuth and Yao 1976).

References

$\endgroup$

1 Answer 1

2
$\begingroup$

I assume that $X$ takes non-negative integral values.


First, suppose that $E X^a<\infty$ for some $a>0$. Then for all $n\ge 0$, it holds either that $$ p_n\log \frac{1}{p_n}\le n^a p_n $$ or $$ p_n\le e^{-n^a}. $$ But in the latter case, we can also bound, say, $\log(1/p_n)\le \sqrt{p_n}$. So both upper bounds are summable, and we get $H(X)<\infty$.


Reciprocally, if we know that $E X^a=\infty$ for all $a>0$, there is not much we can say about $H(X)$. Consider for example $p_n=1/m^2$ if $n=e^m$ for some $m\ge 0$ and $p_n=0$ otherwise. You can check that $p_n$ satisfies the all-infinite-moments condition (in fact, the main term of the sum does not even converge to $0$), but $$ \sum_{n\le e^m} p_n\log(1/p_n)=\sum_{k\le m} \frac{\log(k^2)}{k^2}. $$ So $H(X)<\infty$.

$\endgroup$
5
  • $\begingroup$ Thank you for the response. Can you answer for the case when $X$ can be negative as well as positive? All along I intended the case when $X$ can be positive and/or negative, unless stated otherwise in the question, especially the case when $X$ can take any positive or negative integer. $\endgroup$
    – Peter O.
    Feb 18 at 18:27
  • $\begingroup$ @PeterO. - so what does the moment condition correspond to now? Should we put absolute values? $\endgroup$
    – md5
    Feb 18 at 19:04
  • $\begingroup$ Only $z$th moments for any real $z > 0$, and the absolute values of the moments are taken (e.g., where I said "$\mathbb{E}[X^z] = \infty$", I meant "$\mathbb{E}[X^z]$ is infinite" or "$\mathbb{E}[X^z]$ is positive infinity or negative infinity"). $\endgroup$
    – Peter O.
    Feb 18 at 19:12
  • $\begingroup$ @PeterO. - but this is not even defined if the exponent is not an integer $\endgroup$
    – md5
    Feb 18 at 20:31
  • $\begingroup$ In that case, only one question remains to be asked: If $X$ is supported on the whole set of integers and has a finite mean, then is $H(X)$ finite? $\endgroup$
    – Peter O.
    Feb 18 at 20:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.