A couple of questions related to the following problem (please note that I am not familiar with all the theory behind Lagrange multipliers).

Starting from the definition of entropy $$H(p) = -\int p(x) \ln \left(p(x)\right) \,\mathrm d x$$ derive the general equation for the maximum-entropy distribution given constraints expressed in the general form $\int$ $b_{k}(x)p(x)dx = a_{k}$, $k= 1, 2, \dots, q$ as follows:

(a) Use Lagrange undetermined multipliers $λ_{1}, λ_{2}, ..., λ_{q}$ and derive the synthetic function: $H_{s} = -$ $\int{p(x)(ln(p(x))-\sum_{k=0}^{q}{λ_{k}b_{k}(x)})dx}-\sum_{k=0}^{q}$ ${λ_{k}a_{k}}$ State why we know $a_{0} = 1$ and $b_{0}(x) = 1$ for all $x$.

No problem with this.

(b) Take the derivative of $H_{s}$ with respect to $p(x)$. Equate the integrand to zero, and thereby prove that the minimum-entropy distribution obeys:


where the $q + 1$ parameters are determined by the constraint equations.

2 things I do not understand:

  1. Why is it valid to differentiate wrt $p(x)$? How about the impact of such a differentiation on $b_{k}(x)$ (such an impact is ignored in the solution, which treats $b_{k}(x)$ as a constant)?
  2. Equating the integrand to $0$ is sufficient, but can we tell whether this solution is the only one (the integral might evaluate to 0 despite its integrand being different from 0)?
  • $\begingroup$ It'll be maximum entropy, not minimum! (The function $-p\log p$ is concave, so any local extrema are global maxima.) $\endgroup$ – Dap Mar 29 '18 at 8:06

The method of Lagrange multipliers, when used for convex optimization, gives you a witness that what you've found is an optimum. In this case the witness can be defined pointwise. Define $\phi_x(p)=-p[\ln p-\sum_{k=0}^{q}{λ_{k}b_{k}(x)}]$ as a function $[0,\infty)\to\mathbb R.$ For each fixed $x,$ the function $\phi_x$ is strictly concave, and the particular $p(x)$ you derive in (b) obeys $\phi_x'(p(x))=0,$ so $p(x)$ is the unique maximum of $\phi_x.$ It's valid to differentiate with respect to $p$ because $x$ is fixed.

That $p(x)$ is the unique maximum means $\phi_x(q)<\phi_x(p(x))$ whenever $q\neq p(x).$ Now letting $x$ be a free variable again, it is obvious that any function $q:\mathbb R\to [0,\infty)$ satisfies $\int\phi_x(q)dx\leq \int\phi_x(p)dx,$ with equality only if $q$ is equal to $p$ almost everywhere, or assuming continuity. This is just using the fact that the integral of a non-negative function is non-negative, with equality only for functions that are almost everywhere zero.

To conclude, suppose that $q$ satisfied the same constraints as $p$ - you would find that $\int\phi_x(q)dx- \int\phi_x(p)dx=H(q)-H(p).$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.