I know that certain probability distributions may be derived from the requirement that entropy be maximal along with a constraint such as fixed variance. In the case of fixed variance, for example, one finds the normal distribution. In particular, the maximisation is over the set of all (!) continuous PDFs with that fixed variance.

Now my question is, is there a similarly general derivation of the Poisson distribution as a maximum entropy distribution? E.g. fixing that mean and variance are equal and maximising entropy? I have found a couple of articles but they always seem to prove maximality on a restricted set of discrete PDFs. Is it because there is no more general maximum entropy principle for the Poisson distribution? If so, is it because the discrete case is simply more complex than the continuous one?

  • $\begingroup$ The form of the maximum entropy distribution for the discrete case is provided in the wikipedia article. It does not appear that this form corresponds to the Poisson distribution for some set of constraints (see the Table of the above link). $\endgroup$ – Stelios Apr 19 '17 at 11:57
  • $\begingroup$ Cf. here: arxiv.org/pdf/math/0603647v2.pdf and here: yaroslavvb.com/papers/harremoes-binomial.pdf. In both cases, the authors consider a subset of all discrete PDFs. $\endgroup$ – Cyclone Apr 19 '17 at 19:24
  • $\begingroup$ I thought you were asking about the entropy maximizing distribution out of all discrete PDFs. Clearly, if we restrict our search to a subset of PDFs, the Poisson may indeed be the entropy maximizer. As a trivial example, consider the maximizer from the set of two PDFs: (1) $\mathbb{P}(x=k)=e^{-\lambda}\frac{\lambda^k}{k!}$ (Poisson), and (2) $\mathbb{P}(x=0)=1, \mathbb{P}(x=l)=0, x\neq0$ (certain event). Of course, it is of (theoretical?) interest to find non-trivial sets of distributions for which the Poisson is the maximizer, and that's what these references are addressing. $\endgroup$ – Stelios Apr 19 '17 at 19:41
  • $\begingroup$ Yes you understood my question correctly. Maybe I misunderstand your number (2), but it doesn't look like a PDF to me if it is only non-zero and equal to one at x=0 (not normalised). OK, so you're saying you believe the Poisson or Bernoulli distribution do not have a maximum entropy principle such as the normal distribution. In a way, that seems to indicate for me that they are more complicated things than for example the normal distribution. Why is it that they are more complicated? Is it just that discrete distributions are more tricky than continuous ones? $\endgroup$ – Cyclone Apr 20 '17 at 8:15
  • $\begingroup$ Apologies, it should be $l\neq 0$ (instead of $x\neq 0$) in (2). This is a perfectly valid PDF (although, trivial). However, you can easily replace (2) by some other PDF that has a smaller entropy that the Poisson. I would not say that the discrete distribution space is more complicated, since the maximum entropy distributions are obtained with the same methodology as in the continuous case. It only happens that the result is not Poisson (and I don't see why one should expect the Poisson dist. to be the entropy maximizer). $\endgroup$ – Stelios Apr 20 '17 at 8:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.