# Proof that the distribution that maximizes the differential entropy is the Gaussian; why constrain the first and second moments?

In Section 1.6 of PRML by Bishop, it says, in p53,

In order for this maximum [maximum of differential entropy] to be well defined, it will be necessary to constrain the first and second moments of $$p(x)$$

where the definition of differential entropy of distribution is given by $$H[x] = −\int p(x) \ln p(x) dx$$ Why is it necessary to constrain the first and second moments, but not the higher order moments? I understand moments gives you information on the distribution, so it makes sense to set moments given, but why is it that you don't set the third and higher order moments given? Is that something that you omit for simplicity but could do?

• What is the definition of differential entropy?
– Mark
Nov 12, 2019 at 4:07
• @Mark Editied the question. Thanks! Nov 12, 2019 at 4:15

For distributions which have an expectation, note that the expectation does not contribute to the differential entropy. Pinning down a particular one, say $$\mu=0$$, is at least a mathematical convenience that allows you to start doing actual computations, and to end up with a single distribution in the end, as opposed to a whole set of distributions who are identical except for shifts in their means.
Variance also makes sense as a constraint, since it makes the competition between distributions fair in some sense. It's like saying, for all distributions which are $$\sigma$$ wide, which one has the highest differential entropy?
As for why you don't constrain higher moments, I think you actually could, but you would end up with a different winning distribution. You could ask, for all distributions who are skewed $$\gamma$$ (third moment), which is the one with the highest differential entropy? I think for machine learning applications, you often have an intuition for how wide your data is, but maybe not for how skewed it is, so it may be also an issue of usefulness.