3
$\begingroup$

I know when the mean and variance of $\ln x$ are both fixed, then the maximum entropy probability distribution is lognormal. When the mean of a random variable is fixed the MEPD is the exponential distribution. My question is, what is the MEPD in the continuous case when neither the mean or variance are fixed with support on $[0, \infty)$?

$\endgroup$
  • $\begingroup$ Probably Gaussian noise which has a normal distribution. Does Burg's maximum entropy theorem apply to your case? $\endgroup$ – Wintermute Jun 26 '13 at 17:48
  • $\begingroup$ If you fix nothing, are you searching for the extrema of $H[p]=\int -p(x)\ln p(x)dx-\lambda(\int p(x)dx-1)$? $\endgroup$ – Avitus Jun 26 '13 at 17:48
  • $\begingroup$ I thought you would just be maximizing this: H[p]=-∫p(x)lnp(x)dx but I could very easily be wrong I am not incredibly familiar with the process of obtaining the MEPD $\endgroup$ – user2183336 Jun 26 '13 at 18:25
  • $\begingroup$ I wrote an answer which considers the discrete case: I think it is a good starting point to get some insights. $\endgroup$ – Avitus Jun 26 '13 at 18:36
1
$\begingroup$
  • Discrete case

In the discrete case you need to consider the functional

$$H[p]=-\sum_{i=1}^n p_i \ln(p_i)+\lambda(\sum_{i=1}^n p_i-1)$$

as we consider a single constraint.

Setting $\frac{\partial H[p]}{\partial p_i}=0$ for all $i=1,\dots,n$ we arrive at

$$-\ln(p_i)-1+\lambda=0\Leftrightarrow p_i=e^{\lambda-1}.$$

Imposing $\sum_{i=1}^n p_i-1=0$, one gets

$\lambda=1-\ln(n)$, or $p_i=e^{1-\ln(n)-1}=\frac{1}{n}$.

In summary, the wished distribution is the uniform probability distribution.

  • Continuous case

The continuous case needs more care, due to non trivial integration range. We want to maximize the functional

$$H[p]=-\int_{0}^{\infty}p(x)\ln(p(x))dx+\lambda(\int_{0}^{\infty}p(x)dx-1), $$

where $p$ has support $[0,\infty]$ and $p(0)=p(\infty)=0$. We apply the calculus of variations by considering any distribution $\phi$ s.t. $p(0)=p(\infty)=\phi(0)=\phi(\infty)$. We compute the variation

$$\frac{\delta H}{\delta\phi}|_{p}=\lim_{\epsilon\rightarrow 0} \frac{H[p+\epsilon\phi]-H[p]}{\epsilon}=\lim_{\epsilon\rightarrow 0}\frac{1}{\epsilon}\left[\int_{0}^{\infty}\left(F(p+\epsilon\phi,x)-F(p,x)\right)dx+ \lambda(\int_{0}^{\infty}\epsilon\phi dx)\right],$$

where $F(p,x)=-p(x)\ln(p(x))$ and $F(p+\epsilon\phi,x)=-(p(x)+\epsilon\phi)\ln(p(x)+\epsilon\phi)$.

Using

$$F(p+\epsilon\phi,x)-F(p,x)=\epsilon\phi\frac{\partial F}{\partial p}(p,x)+O(\epsilon^2)$$

we have

$$\frac{dH}{d\phi}|_{p}=\int_{0}^{\infty}\left(\frac{\partial F}{\partial p}(p,x)+\lambda\right)\phi dx$$

where $\frac{\partial F}{\partial p}(p,x)=-\ln(p(x))-1$. In summary

$$-\ln(p(x))-1+\lambda=0 $$

or $p(x)=e^{\lambda-1}$, with $\int_0^{\infty}e^{\lambda-1}dx=1,$ which is not possible.

Roughly speaking, the absence of additional constraints like the fixed mean one

$$\int_{0}^{\infty} xp(x)dx=\mu$$

does not allow to arrive at "more interesting" differential equations for $p(x)$. Note that the $F$ does not depend on $p'(x)$: this leads to the simplified Euler Lagrange equation

$$\frac{\partial F}{\partial p}(p,x)+\lambda=0.$$

$\endgroup$
  • 1
    $\begingroup$ Sorry I should have clarified that I am searching for the continuous case. again my one constraint is the support is [0,+∞). Upvote for the attempt and my miscommunication though. $\endgroup$ – user2183336 Jun 26 '13 at 18:37
  • $\begingroup$ Support of the probability distribution, am I right? I can add some details to my answer. Thanks for the clarification. $\endgroup$ – Avitus Jun 26 '13 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.