1
$\begingroup$

I have a simple optimization problem, but somewhere I am making a mistake. I want to maximize the entropy for the four $p_1, p_2, p_3, p_4$. I want to use Lagrange multiplier with constraints. Thus:

$\sum_i p_i \log_2(p_i)$ for $i$ of the interval $[1,4]$
s.t.
$g(p_1,p_2,p_3,p_4) = p_1+p_2+p_3+p_4 = 1$ .

Now I want to compute it with Lagrange multiplier: $$L(p_1,p_2,p_3,p_4,\lambda)= -p_1 \log p_1 - p_2 \log p_2 - p_3 \log p_3 - p_4 \log p_4- \lambda(p_1 + p_2 + p_3 + p_4 - 1)$$ If I compute the partial derivatives:
$$\frac{\partial L}{\partial p_i} = -\log p_i - 1 - \lambda= 0$$ Thus:
$$p_i = - e - e^{\lambda}$$ If I now insert it in the Lagrange function $L$:
$$L(p_1,p_2,p_3,p_4,\lambda)= -p_1 \log p_1 - p_2 \log p_2 - p_3 \log p_3 - p_4 \log p_4- \lambda(p_1 + p_2 + p_3 + p_4 - 1)$$ I will have negative values in the log:
$$-(- e - e^{\lambda} \log(-e-e^{\lambda}))$$ which is not solvable.
I don't know where my mistake is.

|cite|improve this question
$\endgroup$
  • $\begingroup$ Note that it makes no sense to use Lagrange multipliers for this, because $p$ isn't differentiable (it's non-zero only at four points!). I've asked this question in the past and will link to it. $\endgroup$ – Clarinetist May 1 '18 at 11:45
  • 1
    $\begingroup$ @Clarinetist This problem is defined on a finite set with cardinality 4, not the same as your problem :) $\endgroup$ – Xiangxiang Xu May 1 '18 at 12:07
2
$\begingroup$

Since $\frac{\partial L}{\partial p_i} = -\log p_i - 1 - \lambda = 0$, we have $$p_i = \mathrm{e}^{-\lambda - 1}.$$ As a result, $$p_1 = p_2 = p_3 = p_4 = \frac{1}{4},$$

i.e., the maximum entropy distribution should be the uniform distribution.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.