1
$\begingroup$

In Pattern Recognition and Machine Learning, Bishop states

"Let us now consider the maximum entropy configuration for a continuous variable. In order for this maximum to be well defined, it will be necessary to constrain the first and second moments of p(x) as well as preserving the normalization constraint"

Why is it the case?

Entropy:

continuous entroy

Moments: CE moments

$\endgroup$
1
$\begingroup$

Consider the uniform distribution on the interval $[-x,x], x \ge 1$ . It's entropy is $ln(2x)$ . When x go to infinity , the entropy go to infinity as well, therefore the maximum is not well defined .

$\endgroup$
4
  • $\begingroup$ My question is about the moments, why do you consider these moments to prove that it's well defined $\endgroup$ – ted Feb 23 '19 at 17:38
  • 1
    $\begingroup$ Without the moments bounds , as the exemple shows , the entropy may not by finite . Now consider that the random variable has finite variance . The probability distribution with the highest entropy is the uniform distribution ( we restrict our atention to an interval $[a,b] $ ) . The variance of this RV is $1/12(b-a)^{2} $ and it's entropy is $ln(b-a)$ . Therefore a bound on the variance implies a bound on the entropy . $\endgroup$ – Popescu Claudiu Feb 23 '19 at 19:27
  • $\begingroup$ @PopescuClaudiu But I thought Bishop just showed us that the Gaussian, not the uniform, is the distribution having highest entropy for a fixed variance (and first and zeroth moment)... max entropy for a given variance is given by eqn 1.110. $\endgroup$ – Don Slowik Apr 16 '19 at 22:42
  • $\begingroup$ @Don Slowik The probability distribution with the highest entropy is the uniform distribution when no restrictions are enforced (I only impose the technical condition that the variance is finite). The Gaussian is the distribution having highest entropy for a fixed variance , as you said . There is no contradiction here . $\endgroup$ – Popescu Claudiu Apr 17 '19 at 8:38
2
$\begingroup$

Many continuous distributions have infinite entropy. So it makes sense to impose some restrictions before asking which distribution has the largest entropy. For example, if we just required the zeroth moment be 1: $\int^\infty_{-\infty} p(x)dx=1$, $$\text{H}[\text{Unif}(x|a,b)]=\ln(b-a)\ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{(@PopescuClaudiu)}$$ and $$\text{H}[\mathcal{N}(y|\mu, \sigma^2)]={1\over{2}} \{1+\ln(2\pi \sigma^2)\} \ \ \ \ (\text{Bishop eqn. 1.110})$$ can each be made as large as desired by increasing $b-a$ or $\sigma^2$ respectively.

The continuous distribution having the largest entropy on a fixed domain, $x\in[a,b]$, can be found by maximizing $$-\int^b_a p(x)\ln(p(x))dx + \lambda\bigg(\int^b_a p(x)dx-1\bigg)$$ wrt $p(x)$: $$-\ln p(x)-1+\lambda=0$$ $$p(x)=\exp(\lambda-1)=c$$ $$\int^b_a p(x)dx=1\Rightarrow p(x)={1\over{b-a}}$$ as pointed out in the comment by @PopescuClaudiu.

The continuous distribution having the largest entropy and having first and second moments of $\mu={b+a\over2}$, and $\sigma^2={(b-a)^2\over{12}}$ is $$p(x)={1\over(2\pi\sigma^2)^{1/2}}\exp\left\{-{(x-\mu)^2\over2\sigma^2}\right\}$$ as derived in Bishop PRML p.54. Note that each of these distributions have the same $0^{th}, 1^{st}$ and $2^{nd}$ moments, and the Gaussian has a larger entropy $${1\over{2}} \{1+\ln(2\pi \sigma^2)\}= {1\over{2}} \{1+\ln({\pi\over6})+2\ln(b-a)\}>\ln(b-a)$$

A whole collection of possible constraints and corresponding entropy maximizing distributions is given here.

$\endgroup$
3
  • $\begingroup$ Can u explain in detail that how to get from $-\int^b_a p(x)\ln(p(x))dx + \lambda\bigg(\int^b_a p(x)dx-1\bigg)$ wrt p(x) to $-lnp(x) - 1 + \lambda$? $\endgroup$ – SundayCat Apr 30 '19 at 2:03
  • 1
    $\begingroup$ So this is Calculus of Variations which is described in appendix D of Bishop PRML. Basically, the variation of those integrals wrt small, arbitrary changes $\delta p(x)$ should be zero at an extremum. $\endgroup$ – Don Slowik May 1 '19 at 0:48
  • $\begingroup$ Got it, thanks a lot $\endgroup$ – SundayCat Jun 14 '19 at 19:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.