3
$\begingroup$

What kind of distribution do we get if we constrain the range to be the unit interval and also constrain the mean to be $\alpha$?


If we read this table, we see the following two examples of maximum entropy distributions.

The standard normal distribution is the maximum entropy distribution with mean 0 and variance 1.

Similarly, if we restrict out attention to distributions with bounded support, then the uniform distribution is the maximum entropy distribution.

I'm writing a small library to estimate quantiles in a stream of data and am currently keeping track of the sample minimum, maximum, and mean, but am not using the sample mean to estimate quantiles directly.

So, what kind of distribution do we get if we constrain the range to be the unit interval and also constrain the mean to be $\alpha$?

In general, is there a good strategy for figuring out a closed form for the pdf of a maximum entropy distribution satisfying an arbitrary collection of properties?

$\endgroup$
4
  • 1
    $\begingroup$ For each $\alpha \in (0, 1)$, let $c$ solve the equation $\alpha = 1 - \frac{1}{c} + \frac{1}{e^c - 1}$. (Such $c$ uniquely exists.) Then the distribution with the following p.d.f. $$ f_{\alpha}(x) = \frac{c}{e^c - 1} e^{cx} \mathbf{1}_{[0,1]}(x) $$ maximizes the entropy over the space of continuous distributions supported on $[0, 1]$ and has mean $\alpha$. This follows from variational argument. $\endgroup$ Feb 22, 2019 at 23:27
  • $\begingroup$ @SangchulLee, there must be something I'm missing ... but I think that $f_{0.5}$ should be just the uniform distribution since the mean constraint is redundant in the case. $f_{\alpha}$ does not appear to be a constant times an indicator function when $\alpha = 0.5$ . $\endgroup$ Feb 22, 2019 at 23:36
  • 1
    $\begingroup$ That's a good point. As $\alpha \to \frac{1}{2}$, we can prove that $c \to 0$. In such case, $\frac{c}{e^c - 1} \to 1$, so this indeed reduces to the uniform distribution. $\endgroup$ Feb 22, 2019 at 23:37
  • $\begingroup$ @SangchulLee, want to write that as an answer, so we can upvote it? $\endgroup$
    – D.W.
    Jun 29, 2019 at 23:46

1 Answer 1

2
$\begingroup$

Let $\alpha \in (0, 1)$ and $\mathcal{A}$ denote the set of all $f \in L^1([0,1])$ with the constraints

$$f \geq 0, \qquad \int_{0}^{1} f(x) \, \mathrm{d}x = 1, \qquad \int_{0}^{1} x f(x) \, \mathrm{d}x = \alpha. \tag{*} $$

We claim that the differential entropy $h(\cdot)$ is maximized over $\mathcal{A}$ by

$$f_{\max}(x) = e^{ax+b} \mathbf{1}_{[0,1]}(x)$$

for some $a, b \in \mathbb{R}$. (Note that such $a$ and $b$ are uniquely determined from $\text{(*)}$.) Indeed, for any $f \in \mathcal{A}$, the Kullback-Leibler divergence $D(f||f_{\max})$ is non-negative, and so, we get

$$ 0 \leq D(f||f_{\max}) = \int_{0}^{1} f(x) \log \left(\frac{f(x)}{e^{ax+b}}\right) \, \mathrm{d}x = -h(f) - (a\alpha + b). $$

On the other hand, by a direct computation we get $h(f_{\max}) = -(a\alpha + b)$. So it follows that

$$ h(f) \leq h(f_{\max}) $$

for all $f \in \mathcal{A}$. This proves that $f_{\max}$ is a maximizer of $h(\cdot)$ over $\mathcal{A}$. (Together with the fact that $h(\cdot)$ is strictly concave, it is easy to prove that $f_{\max}$ is the unique maximizer up to modification on null sets.)


For the reference, see Theorem 5.1 of Keith Conrad. Probability Distributions and Maximum Entropy.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.