Given a multinomial distribution:

$Mul(\overline{x}|n, \overline{p}) = {n \choose x_{1...k}!}\prod_{i}^{k}p_i^{x_i} $

And the entropy of a probability distribution:

$Entropy(x) = -\sum_i^n p_x(x)log(p_x(x))$

How do we maximize entropy when it comes to multinomial w.r.t $\overline{p}$? It's simple if we have a binomial example, which is p and 1-p, and just solving for p when the derivative of the entropy with respect to p is equal to 0(or if we can't find a maxima, then use lagrange multipliers with the bound shown next. However, in this case we have to satisfy a bound on maximizing entropy:

$\sum_i^k p_i = 1$

Any hints would be appreciated!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.