For a real-valued, 1-dimensional, continuous random variable $X$ with density $f(x)$, I am trying to determine if maximizing the Shannon differential entropy of $f(x)$ is mathematically equivalent to finding the $f(x)$ with the least predictive density (as defined below) for a given set of constraints:

Least predictive density: The density that minimizes the maximum probabiliy associated with each value of the Lebesgue measure on the domain of $f(x).$

For example, let $X$ be an RV with domain ($-\infty $,$+\infty $) and specified mean $M$ and variance $V$ and $I$ be a set of intervals on the real line with total Lebesgue measure $\mathcal{L}$ (i.e., sum of the measure of each interval in the set. The actual intervals not specified, only their total Lebesgue measure is given). The least predictive distribution would minimize the maximum probability that can be contained in I, for each value of L, while also having mean M and variance V.

I don't know if this problem can be formulated generally as a functional so that calculus of variations can be applied. At this point, I have formulated it as a functional for the specific case of a fixed interval for unimodal symmetric unbounded densities, but generalizing this has been problematic.

Related to the above, this problem appears to boil down to minimizing the maximum value of the density subject to any constraints (such as specified mean and variance).

Given the above "preamble", the two issues I am hoping to get guidance on are:

1) Is maximum Shannon entropy equivalent to minimal predictability (as defined above for a particular set of constraints)?

2) Is minimal predictability equivalent to minimizing the maximum density (subject to a set of constraints)?

I hope I have specified this precisely enough and given enough background on my current thinking and work on this. Any pointers, references, or research results related to the above would be much appreciated.

Thanks :)

  • 1
    $\begingroup$ I hope to see comprehensive answer to this question as I found a reference that links maximizing Shannon Information to minimizing D-optimal criteria $\endgroup$ – GENIVI-LEARNER Mar 29 at 17:50
  • $\begingroup$ @GENIVI-LEARNER please, what is the reference? $\endgroup$ – Daniel S. Jul 24 at 14:40

You probably already know this, but maximizing differential entropy results in MaxEnt distributions which are exponential family (derived via variational calculus). For the 1D example of mean and variance constraint, it gives the Gaussian distribution, while your idea of constraining the max probability would give uniform distribution. I may have not understand your claim correctly?

| cite | improve this answer | |
  • $\begingroup$ where can I see the fully-derived maximization of differential entropy resulting in an exponential class distribution? $\endgroup$ – develarist Aug 24 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.