2
$\begingroup$

In Section 1.6 of PRML by Bishop, it says, in p53,

In order for this maximum [maximum of differential entropy] to be well defined, it will be necessary to constrain the first and second moments of $p(x)$

where the definition of differential entropy of distribution is given by $$ H[x] = −\int p(x) \ln p(x) dx $$ Why is it necessary to constrain the first and second moments, but not the higher order moments? I understand moments gives you information on the distribution, so it makes sense to set moments given, but why is it that you don't set the third and higher order moments given? Is that something that you omit for simplicity but could do?

enter image description here enter image description here

$\endgroup$
2
  • 2
    $\begingroup$ What is the definition of differential entropy? $\endgroup$
    – Mark
    Nov 12, 2019 at 4:07
  • $\begingroup$ @Mark Editied the question. Thanks! $\endgroup$
    – alicec
    Nov 12, 2019 at 4:15

1 Answer 1

2
$\begingroup$

If some sort of grouping isn't done, then it doesn't really make sense to talk about a distribution with the highest differential entropy in the same way it isn't possible to talk about a rectangle with the highest area. You could just keep picking bigger and bigger rectangles. But you could sensibly ask, what is the rectangle with highest area, whose perimeter is 1.

The book's remark is not to say that differential entropy itself requires mean and variance to be defined, because the Cauchy distribution has a differential entropy without having a mean and variance. It's only saying that in order to define a maximum, you have to constrain the problem somehow.

For distributions which have an expectation, note that the expectation does not contribute to the differential entropy. Pinning down a particular one, say $\mu=0$, is at least a mathematical convenience that allows you to start doing actual computations, and to end up with a single distribution in the end, as opposed to a whole set of distributions who are identical except for shifts in their means.

Variance also makes sense as a constraint, since it makes the competition between distributions fair in some sense. It's like saying, for all distributions which are $\sigma$ wide, which one has the highest differential entropy?

As for why you don't constrain higher moments, I think you actually could, but you would end up with a different winning distribution. You could ask, for all distributions who are skewed $\gamma$ (third moment), which is the one with the highest differential entropy? I think for machine learning applications, you often have an intuition for how wide your data is, but maybe not for how skewed it is, so it may be also an issue of usefulness.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .