Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Consider a random variable $X$ taking values over $\mathbb{N}$. Let $\mathbb{P}(X = i) = p_i$ for $i \in \mathbb{N}$. The entropy of $X$ is defined by $$H(X) = \sum_i -p_i \log p_i.$$ Is it possible for $H(X)$ to be infinite?

share|cite|improve this question
Googling "random variable infinite entropy" gives this example. – David Mitra Jan 15 '13 at 15:00
@David: You could turn that into a full answer by quoting the distribution. – joriki Jan 15 '13 at 15:02
up vote 7 down vote accepted

Yes, it can.

In this pdf, it is shown that the following random variable $X$ has infinite entropy:

Let $X$ take the values $2,3,\ldots$ and define $$p_n={1\over A n\log^2_2 n},\ \ \ n\ge2,$$ where $A = \sum\limits_{n=2}^\infty {1\over n\log^2_2 n}$ ($A$ is a convergent sum by the Integral Test). Then $X$ with the probability mass function $P[X=n]=p_n$, $n\ge 2$, has infinite entropy.

For completeness, I'll reproduce the argument contained in the link above that $H(X)=\infty$ here:

We have $$\eqalign{ H(X)&=-\sum_{n=2} p_n \log_2 p_n \cr &=\sum_{n=2}^\infty\bigl( -\log_2(p_n)\bigr) p_n\cr &= \sum_{n=2}{\log_2( An \log_2^2 n )\over A n\log_2^2 n} \cr &=\sum_{n=2}^\infty {{\log_2 A+\log_2 n +2\log_2(\log_2 n))}\over A n \log_2^2 n}\cr &=\sum_{n=2}^\infty\biggl[\ \color{maroon}{\log_2 A\over A n\log_2^2 n}+\color{darkgreen}{1\over A n\log_2 n}+\color{darkblue}{2\log_2(\log_2 n)) \over A n\log_2^2 n}\ \biggr].\cr } $$

$$ \color{maroon}{ \sum\limits_{n=2}^\infty {\log_2 A\over A n\log_2^2 n}} ={\log_2 A\over A} \sum\limits_{n=2}^\infty {1\over n\log_2^2 n} ={\log_2 A\over A}\cdot A=\log_2 A. $$

and $$ \color{darkblue}{ \sum\limits_{n=2}^\infty {2\log_2(\log_2 n)\over A n\log_2^2 n}} $$ is a sum consisting of nonnegative terms.

It will follow that $H(X)=\infty$, if we can show that the sum $$ \color{darkgreen}{\sum_{n=2}^\infty {1\over An\log_2 n}} $$ diverges to $\infty$. But, this follows easily from the Integral Test.

(From the Integral Test, it follows that $\int_2^\infty {dx\over x\log_2^p x}$ converges if and only if $p>1$.)

share|cite|improve this answer
Thanks! I wonder if there are sufficient (or even necessary and sufficient) conditions for entropy to be finite. – VSJ Jan 15 '13 at 15:14

Consider the independent variables $X_0, X_1, X_2 , X_3 \cdots$, where $X_k$ have non overlapping discrete uniform distributions over $2^k$ values, eg $X_0=1$, $X_1: U[2\ldotp\ldotp3]$, $X_2: U[4\ldotp\ldotp7]$, $X_3:U[8\ldotp\ldotp15]$, etc. The respective entropies are $H(X_k) = k$ bits.

Let $Y$ be generated by picking $X_k$ with probabilities $m_k\ge 0$ ($\sum_{k=0}^\infty m_k=1)$. This is known as a mixture, with mixing distribution $m_k$, and the resulting entropy (provided the components do not overlap) is

$$\begin{array}{} H(Y)&=&H(m)+m_0 H(X_0)+m_1 H(X_1)+ m_2 H(X_2) + m_3 H(X_3) +\cdots \\ &=& H(m) + m_1 + 2 \, m_2 + 3 \, m_3 \cdots \\&=& H(m) + \sum k \, m_k \end{array}$$

Hence, by chosing any mixing distribution $m_k$ such that $\sum k \, m_k=\infty$, we attain infinite entropy. For example: $m_k=A/k^2$. The example in David's answer essentially corresponds to this choice.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.