# How to interpret that a distribution does not have defined entropy (or has infinite entropy)?

An entropy (is Shannon sense) can be interpreted as uncertainty or missing knowledge. When the knowledge is added, the entropy decreases. Hence it can also be interpreted as information content.

However, there are discrete distributions which do not have an entropy because $$-\sum_{i=1}^{\infty}p_i\log p_i$$ tends to infinity (see here).

How to interpret this situation? Does non-existent or infinite entropy means that it is not possible to remove uncertainty and get complete knowledge about values of variable described by such distribution?

• If you interpret the entropy as being the expected minimal amount of number of yes/no questions, you may get a feeling of what is happening. – P. Quinton Feb 24 at 15:24

I think that the most natural example of infinite entropy is a continuous distribution (say, uniform on $$[0,1]$$), which can be regarded as the limit of a discrete uniform distribution for increasingly large number of values. It's not surprising that a real number in $$[0,1]$$ has infinite entropy, because to describe it you need an infinite number of bits (think of its binary fractional representation). Put in other way, in a real number in $$[0,1]$$ you can code an arbitrarily big amount of information.
• I think there is a slight misunderstanding. In case of continuous distribution, the entropy is calculated as $-\int_a^b f(x)\ln f(x)dx$. For continuous uniform distribution $f(x) = \frac{1}{b-a}$. Since $f(x)$ is constant and therefore $\ln f(x)$ is also constant, entropy is $\ln(b-a)$. This is finite number. – Martin Vesely Feb 25 at 8:30