Does any probability distribution have an entropy defined?

Question

There are lot of probability distribution having infinite moments, for example Cauchy distribution has even the first moment infinite. Therefore, often we cannot calculate second moments used in measures of dispersion or uncertainty (variation and standard deviation). In some application, for example finance, we could replace such measures with entropy. However, firstly I would like to know whether any probability distribution has defined entropy (in particular Shannon entropy).

In case of discrete distributions with finite number of possible outcome the entropy is $$ H = -\sum_{i=1}^{n} p_i \log p_i, $$ where $p_i$ is probability of ith outcome. Since $p_i > 0$ and number of terms in the sum is finite, the sum is defined and it is finite.

But I am getting stuck with case $n \rightarrow +\infty$. In this case I need to prove that the sum under condition that $\sum_{i=1}^{+\infty}p_i = 1$ converges.

Similarly in case of continuous distribution, I would need to prove that integral $$ H = -\int_{\mathbb{R}} f(x) \log f(x) \mathrm{d}x $$ for any real function satisfying $f(x) > 0\,\, \forall \in \mathbb{R}$ and $\int_\mathbb{R}f(x)\mathrm{d}x=1$ exists and it is finite.

Is it possible to prove these statements?

Answer 1

It is possible to have infinite Shannon entropy when dealing with countably many outcomes. For example, $$ p_n = \frac{c}{n (\log n)^{1+\epsilon}}\quad n\geq 3 $$ where the constant $c=1/\sum_{n\geq 3}\frac1{n(\log n)^{1+\epsilon}}$ is chosen to satisfy $\sum_{n=3}^\infty p_n=1$. Since $\sum\frac1{n(\log n)^{1+\epsilon}}$ converges only for $\epsilon>0$ and $$ -p_n\log p_n=\frac{c}{n(\log n)^{1+\epsilon}}(-\log c+\log n+(1+\epsilon)\log\log n) \sim \frac{c}{n(\log n)^\epsilon} $$ we see that for $0<\epsilon\leq 1$ the sum $H=-\sum_n p_n\log p_n$ diverges.

Answer 2

The result that you want looks very unlikely, and in fact one can prove that it is false by the following argument: let ${\varphi} \left(x\right) =-x \ln \left(x\right)$. For ${\alpha} \in \left(0 , 1\right)$, let us take $n$ numbers ${u}_{1} , \ldots , {u}_{n}$, all equal to $\frac{{\alpha}}{n}$. We have

\begin{equation} \sum _{i = 1}^{n} {u}_{i} = {\alpha} \qquad \sum _{i = 1}^{n} {\varphi} \left({u}_{i}\right) =-{\alpha} \ln \left(\frac{{\alpha}}{n}\right) = {\varphi} \left({\alpha}\right)+{\alpha} \ln \left(n\right) \end{equation}

By choosing $n \geqslant {e}^{1/{\alpha}}$, we can ensure that $\sum _{i = 1}^{n} {\varphi} \left({u}_{i}\right) \geqslant 1$.

Let us now consider the sequence ${{\alpha}}_{k} = {2}^{{-k}}$ for $k \geqslant 1$. let ${n}_{k} = \left\lceil {e}^{1/{{\alpha}}_{k}}\right\rceil $ and let us choose the sequence ${p}_{i}$ by the following algorithm: the ${n}_{1}$ first terms are equal to ${{\alpha}}_{1}/{n}_{1}$, then the ${n}_{2}$ next terms are equal to ${{\alpha}}_{2}/{n}_{2}$, etc.

It follows from this construction that

\begin{equation} \sum _{i = 1}^{\infty } {p}_{i} = 1 \qquad \sum _{i = 1}^{\infty } {\varphi} \left({p}_{i}\right) = \infty \end{equation}