# Can the entropy of a random variable with countably many outcomes be infinite?

Consider a random variable $X$ taking values over $\mathbb{N}$. Let $\mathbb{P}(X = i) = p_i$ for $i \in \mathbb{N}$. The entropy of $X$ is defined by $$H(X) = \sum_i -p_i \log p_i.$$ Is it possible for $H(X)$ to be infinite?

-
Googling "random variable infinite entropy" gives this example. – David Mitra Jan 15 '13 at 15:00
@David: You could turn that into a full answer by quoting the distribution. – joriki Jan 15 '13 at 15:02

Yes, it can.

In this pdf, it is shown that the following random variable $X$ has infinite entropy:

Let $X$ take the values $2,3,\ldots$ and define $$p_n={1\over A n\log^2_2 n},\ \ \ n\ge2,$$ where $A = \sum\limits_{n=2}^\infty {1\over n\log^2_2 n}$ ($A$ is a convergent sum by the Integral Test). Then $X$ with the probability mass function $P[X=n]=p_n$, $n\ge 2$, has infinite entropy.

For completeness, I'll reproduce the argument contained in the link above that $H(X)=\infty$ here:

We have \eqalign{ H(X)&=-\sum_{n=2} p_n \log_2 p_n \cr &=\sum_{n=2}^\infty\bigl( -\log_2(p_n)\bigr) p_n\cr &= \sum_{n=2}{\log_2( An \log_2^2 n )\over A n\log_2^2 n} \cr &=\sum_{n=2}^\infty {{\log_2 A+\log_2 n +2\log_2(\log_2 n))}\over A n \log_2^2 n}\cr &=\sum_{n=2}^\infty\biggl[\ \color{maroon}{\log_2 A\over A n\log_2^2 n}+\color{darkgreen}{1\over A n\log_2 n}+\color{darkblue}{2\log_2(\log_2 n)) \over A n\log_2^2 n}\ \biggr].\cr }

Now
$$\color{maroon}{ \sum\limits_{n=2}^\infty {\log_2 A\over A n\log_2^2 n}} ={\log_2 A\over A} \sum\limits_{n=2}^\infty {1\over n\log_2^2 n} ={\log_2 A\over A}\cdot A=\log_2 A.$$

and $$\color{darkblue}{ \sum\limits_{n=2}^\infty {2\log_2(\log_2 n)\over A n\log_2^2 n}}$$ is a sum consisting of nonnegative terms.

It will follow that $H(X)=\infty$, if we can show that the sum $$\color{darkgreen}{\sum_{n=2}^\infty {1\over An\log_2 n}}$$ diverges to $\infty$. But, this follows easily from the Integral Test.

(From the Integral Test, it follows that $\int_2^\infty {dx\over x\log_2^p x}$ converges if and only if $p>1$.)

-
Thanks! I wonder if there are sufficient (or even necessary and sufficient) conditions for entropy to be finite. – VSJ Jan 15 '13 at 15:14

Consider the independent variables $X_0, X_1, X_2 , X_3 \cdots$, where $X_k$ have non overlapping discrete uniform distributions over $2^k$ values, eg $X_0=1$, $X_1: U[2\ldotp\ldotp3]$, $X_2: U[4\ldotp\ldotp7]$, $X_3:U[8\ldotp\ldotp15]$, etc. The respective entropies are $H(X_k) = k$ bits.

Let $Y$ be generated by picking $X_k$ with probabilities $m_k\ge 0$ ($\sum_{k=0}^\infty m_k=1)$. This is known as a mixture, with mixing distribution $m_k$, and the resulting entropy (provided the components do not overlap) is

$$\begin{array}{} H(Y)&=&H(m)+m_0 H(X_0)+m_1 H(X_1)+ m_2 H(X_2) + m_3 H(X_3) +\cdots \\ &=& H(m) + m_1 + 2 \, m_2 + 3 \, m_3 \cdots \\&=& H(m) + \sum k \, m_k \end{array}$$

Hence, by chosing any mixing distribution $m_k$ such that $\sum k \, m_k=\infty$, we attain infinite entropy. For example: $m_k=A/k^2$. The example in David's answer essentially corresponds to this choice.

-