I'll preface this by saying that I haven't taken an in depth study of the calculus of variations and have only come across it recently in applications; in depth study is on my to do list.

I'm wondering if there is any sense in which the following heuristic for functional optimization is valid. I'm interested in finding a probability density $f$ on $[0, 1]$ with maximum entropy $H[f] = - \int f \log f \ dx$, subject to the constraint that $\int x f(x) \ dx = \mu$. I set this up via Lagrange multipliers as $$ \arg \max_f \ -\int f \log f \ dx - \lambda_1 \left(\int f \ dx - 1\right) - \lambda_2 \left(\int x f \ dx - \mu\right). $$ Now, instead I solve an approximate problem using Riemann sums; we partition $[0, 1]$ with $0 = x_0 < x_1 < \cdots < x_n = 1$ and approximate this by $$ \arg \max_f \ -\sum f_{x_j} \log f_{x_j} \Delta_{x_j} - \lambda_1 \left(\sum f_{x_j} \Delta_{x_j} - 1\right) - \lambda_2 \left(\sum x_{j} f_{x_j} \ \Delta_{x_j} - \mu \right). $$ This can be solved via multivariate calculus and leads to $f(x) = a e^{bx}$, a truncated exponential distribution, as the density on $[0, 1]$ with maximal entropy for a fixed mean, which I believe is the correct answer.

There are a few problems right off the bat. This makes the assumption of Riemann integrability, which is undesirable. It also ignores measure theoretic details (e.g. the solution to the optimization should only be unique up to a.e. $dx$ equivalence classes). But perhaps if I constrain the optimization to being over continuous functions, this provides an argument which can be patched up and made rigorous?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.