3
$\begingroup$

I know that the maximum entropy distribution with over the non-negative integers fixed mean is a geometric distributions.

However, I cannot find conclusive information about what are the maximum entropy distributions over the whole integers (negative + non-negatives).

As per the comment below, in this case, we need at least to specify both the mean and the standard deviation of the distribution for the "maximum entropy" criterion to make sense.

It is well known that if we consider the whole real line, the maximum entropy distribution with a given mean and variance is a Gaussian normal distribution.

My questions are therefore: 1- For a given mean and variance, is there a maximum entropy distribution over the integers? 2- If it exists, can it be expressed in a closed form, like gaussian or geometric distributions?

My guess is that the answer is "yes" for 1) and "no" for 2) (since I cannot find any mention of a closed form anywhere). But I would be very happy to see a confirmation of this.

EDIT: Since it is now clear that the variance needs to be specified for the "maximum entropy" criterion to make sense, I rewrote the question accordingly.

$\endgroup$
  • 1
    $\begingroup$ There is no maximum entropy distribution in this case. You can always increase the entropy by spreading out the probability more, as long as you do it symmetrically about the mean. $\endgroup$ – Daniel Weissman Apr 1 '15 at 22:43
  • $\begingroup$ Thank you. I was actually more or less expecting that. But then if we specify the variance as well, there should be a maximum entropy distribution, right? (like for the normal distribution in the real case). And if so, do we have a closed form for it? $\endgroup$ – Fabien C. Apr 2 '15 at 3:49
6
$\begingroup$

Solving the Lagrange Equations, we get that the maximum entropy distribution with mean $0$ and variance $1$ is where $$ \sum_{k\in\mathbb{Z}}(k^2-1)e^{-ak^2}=0 $$ which is $a\doteq0.4999998943842821\sim\frac12$. We need to compute the coefficient where $$ c\sum_{k\in\mathbb{Z}}e^{-ak^2}=1 $$ which is $c\doteq0.3989422361322933\sim0.3989422804014327=\frac1{\sqrt{2\pi}}$.

Thus, the maximum entropy distribution on the integers that has a mean of $0$ and variance of $1$, is $$ p_k=c\,e^{-ak^2} $$ where $a$ and $c$ are given above. These values are extremely close to the Gaussian, which has the maximum entropy for a continuous distribution with the same constraints.


Although the function derived above is very close to the Gaussian distribution restricted to $\mathbb{Z}$, $\frac1{\sqrt{2\pi}}e^{-n^2/2}$ is not a probability measure on $\mathbb{Z}$. In fact, the Poisson Summation Formula says that $$ \begin{align} \frac1{\sqrt{2\pi}}\sum_{n\in\mathbb{Z}}e^{-n^2/2} &=1+2\sum_{n=1}^\infty e^{-2\pi^2n^2}\\ &\gt1 \end{align} $$

$\endgroup$
  • $\begingroup$ Hi, thanks for the answer. Sorry for reacting to it a bit late. I was not checking here for a while. I will gladly accept your answer since you are the only one that proposed one in 4 months anyway :-). However it would be nice if you could explain a bit more how you computed the values a and c. $\endgroup$ – Fabien C. Aug 3 '15 at 12:02
  • $\begingroup$ Furthermore, I actually found this pdf (inf.fu-berlin.de/inst/ag-ki/rojas_home/documents/tutorials/…), in which the last section ("Gaussian Maximize Entropy") seem to prove that the maximum entropy form for the discrete case is actually the same as for the continuous case. ie., a = 1/2 and c = 1/sqrt(2pi) . Could it be that you solved numerically a and c and that the differences between your values and the "gaussian" values are only approximation errors ? $\endgroup$ – Fabien C. Aug 3 '15 at 12:06
  • $\begingroup$ The standard Gaussian function, $\frac1{\sqrt{2\pi}}e^{-x^2/2}$, is not even a probability measure on $\mathbb{Z}$ because $$\frac1{\sqrt{2\pi}}\sum_{k\in\mathbb{Z}}e^{-k^2/2}=1.000000005350576$$ so the reference you cite cannot be correct. The correct answer, given above, is very close, but not exactly the standard Gaussian function on $\mathbb{Z}$. $\endgroup$ – robjohn Aug 3 '15 at 14:20
  • $\begingroup$ Regarding the computation of $a$ and $c$, I simply used numerical methods to solve $$\sum_{k\in\mathbb{Z}}(k^2-1)e^{-ak^2}=0$$ for $a$. Once we have $a$, it is a simple division to find the $c$ so that $$c\sum_{k\in\mathbb{Z}}e^{-ak^2}=1$$ $\endgroup$ – robjohn Aug 3 '15 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.