5
$\begingroup$

In my earlier question I asked about a technical aspect of solving a system of equations arising from looking for an entropy-maximizing distribution $p(x)$ continuous on $\mathbb{R}$ and constrained by KL-divergence with a zero-mean Gaussian distribution. That is, in addition to the usual probability density and variance constraints, I have the following constraint for $p(x)$:

$$D(p_N(y)\|p(y))=\int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi}\sigma_N}e^{-y^2/2\sigma^2}\log\frac{\frac{1}{\sqrt{2\pi}\sigma}e^{-y^2/2\sigma^2}}{p(y)}dy<\epsilon$$

Thanks to user anon, the form of the function $p(y)$ was found, but it is not a density function and now I am trying to interpret why is this the case.

First, here is the system of equations (copied from the earlier question) that I derived using Calculus of Variations (and help from Gallager's "Information Theory and Reliable Communication":

$$\begin{align} 0&=\log(p(y))+1-\lambda-\gamma y^2-\eta \left(\frac{e^{-y^2/2}}{\sqrt{2\pi}}\right)\left(\frac{1}{p(y)}\right)\\ 0&=1-\int_{-\infty}^{\infty}p(y)dy\\ 0&=1-\int_{-\infty}^{\infty}y^2p(y)dy\\ 0&=c+\int_{-\infty}^{\infty}\frac{e^{-y^2/2}}{\sqrt{2\pi}}\log(p(y))dy \end{align} $$

(for simplicity I set $\sigma=1$; $c=\epsilon+\frac{1}{2}\log(2\pi )$)

From anon's helpful comment, we can actually solve the first equation in terms of Lambert W function to obtain the following:

$$p(x)=\frac{\eta e^{-y^2/2}}{\sqrt{2\pi}W(e^{-(1+2\gamma)y^2/2+(1-\lambda)})}$$

When $|y|\rightarrow\infty$, $e^{-ay^2+b}\rightarrow 0$, and since $W(0)=0$, $p(y)\rightarrow\infty$. Thus, this is obviously not a pdf!

This is entirely due to the KL-divergence constraint (very similar situation arises when variance constraint is removed). How does one explain this? There are obviously probability distributions that meet the KL-divergence constraint (e.g. a Gaussian with appropriately picked variance). Does this mean that optimal distribution does not exist, and all distributions one can try would be sub-optimal? Is there a rigorous explanation for this?

Perhaps I did something wrong? Is there another method I should have employed?

$\endgroup$
1
  • 3
    $\begingroup$ Stupid question: is $1 + 2\gamma$ necessarily positive? $\endgroup$
    – user856
    Commented Sep 22, 2011 at 5:21

1 Answer 1

3
$\begingroup$

You established the indeterminate form $0/0$; from this you cannot casually conclude $p\to\infty$. In the last line of my comment that you cite, I used the trick $e^{W(t)}=t/W(t)$ in order to simplify the expression - but if you want to look at it asymptotically, you can also analyze the prior form: $$p(y)=\exp(-\alpha+W(e^{\alpha}\beta)),$$ where $\alpha=1-\lambda-\gamma y^2$ and $\beta=\eta e^{-y^2/2}/\sqrt{2\pi}$. Now if $\gamma\in[-1/2,0)$, we get the form $p=e^{-\infty+0}$, so we find that $p\to0$. For $\gamma<-1/2$, you'll have to find a way to show $$W(ae^{-(1/2+\gamma)y^2})+\gamma y^2\to-\infty \text{ as } |y|\to\infty.$$ Above $a>0$ is arbitrary (and $\eta e^{1-\lambda}/\sqrt{2\pi}$ in particular). Note $W(x)>1$ for $x>e$, so we have $$e^W\le W e^W =x$$ hence $W\le\log x$ for sufficiently large $x$, which makes the above $-O(y^2)\to-\infty$ (allowing notational sloppiness), proving $p\to0$ as $|y|\to\infty$ whenever we have $\gamma<0$.


This only addresses your claim that $p$ doesn't vanish at the extremes. For your broader questions concerning maximum entropy, KL divergence, or the original optimization problem, I really have no idea.

$\endgroup$
5
  • $\begingroup$ Full disclosure: I've no love for the tendency to "simplify" $\exp(W(t))$ into $t/W(t)$ myself; the latter has a completely unnecessary removable discontinuity among other things... $\endgroup$ Commented Sep 22, 2011 at 8:02
  • $\begingroup$ @J.M.: Hm, I suppose you're right. I don't work with $W$ often so when I originally commented it seemed like the form with no function composition would be "simpler." $\endgroup$
    – anon
    Commented Sep 22, 2011 at 8:16
  • $\begingroup$ It's just my opinion, of course. Maybe there's a situation when $t/W(t)$ is a better form, but I haven't seen one. $\endgroup$ Commented Sep 22, 2011 at 8:19
  • $\begingroup$ Sorry, I had a family emergency of sorts and haven't been able to reply here. First of all, thank you again, anon. I am appreciative of your time -- you have really elucidated this for me. $\endgroup$
    – M.B.M.
    Commented Sep 27, 2011 at 4:26
  • $\begingroup$ Would you like to take a look at my question based on this problem math.stackexchange.com/q/4503894/64809? $\endgroup$
    – Hans
    Commented Aug 1, 2022 at 21:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .