2
$\begingroup$

An entropy (is Shannon sense) can be interpreted as uncertainty or missing knowledge. When the knowledge is added, the entropy decreases. Hence it can also be interpreted as information content.

However, there are discrete distributions which do not have an entropy because $-\sum_{i=1}^{\infty}p_i\log p_i$ tends to infinity (see here).

How to interpret this situation? Does non-existent or infinite entropy means that it is not possible to remove uncertainty and get complete knowledge about values of variable described by such distribution?

$\endgroup$
1
  • 1
    $\begingroup$ If you interpret the entropy as being the expected minimal amount of number of yes/no questions, you may get a feeling of what is happening. $\endgroup$ – P. Quinton Feb 24 at 15:24
2
$\begingroup$

A discrete distribution (with infinite outcomes) can have an infinite entropy. I don't see much to interpret here. Actually, one might argue that, on the contrary, the surprising thing is that some (most) discrete distributions with infinite outcomes have finite entropy, i.e. they produce a finite amount of information, so they can be described (in average) with a finite number of bits.

I think that the most natural example of infinite entropy is a continuous distribution (say, uniform on $[0,1]$), which can be regarded as the limit of a discrete uniform distribution for increasingly large number of values. It's not surprising that a real number in $[0,1]$ has infinite entropy, because to describe it you need an infinite number of bits (think of its binary fractional representation). Put in other way, in a real number in $[0,1]$ you can code an arbitrarily big amount of information.

$\endgroup$
3
  • $\begingroup$ I think there is a slight misunderstanding. In case of continuous distribution, the entropy is calculated as $-\int_a^b f(x)\ln f(x)dx$. For continuous uniform distribution $f(x) = \frac{1}{b-a}$. Since $f(x)$ is constant and therefore $\ln f(x)$ is also constant, entropy is $\ln(b-a)$. This is finite number. $\endgroup$ – Martin Vesely Feb 25 at 8:30
  • $\begingroup$ @MartinVesely No, that's the differential entropy, which is very different from the Shannon entropy (it doesn't correspond to the average amount of bits of information, or the average bits needed to encode it - and it can be zero or even negative). math.stackexchange.com/questions/2552895/… math.stackexchange.com/questions/2880612/… $\endgroup$ – leonbloy Feb 25 at 10:34
  • $\begingroup$ Thanks. I clarified my question. I am asking about discrete distributions. $\endgroup$ – Martin Vesely Feb 26 at 7:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.