3
$\begingroup$

I have a random variable $X$ and want to show that having an entropy $$ H(X) = - \sum_{i=1}^n p_i \text{log}(p_i) = \text{log}(n)$$ is equivalent to the distribution of $X$ being uniform.

Starting with the distribution is straightforward but I don't see how I can deduce the other implication.

$\endgroup$
1
5
$\begingroup$

Use Jensen's inequality with $f(t) = \log t$. $$H(X) = \sum_{i=1}^n p_i \log \frac{1}{p_i}\leq \log\left(\sum_{i=1}^n p_i \frac{1}{p_i}\right) = \log n$$ $\log t$ is strictly concave, so equality requires $p_i = 1/n$, for all $i$, (in other words, a uniform distribution).

$\endgroup$
3
$\begingroup$

Consider $e^{H(x)} = \prod (\frac{1}{p_i})^{p_i}$

By weighted AM-GM Inequality

$\prod (\frac{1}{p_i})^{p_i} \leq \sum \frac{1}{p_i} \times p_i$ $\implies$ $\prod (\frac{1}{p_i})^{p_i} \leq n$

For the equality to hold: $\frac{1}{p_i} = \frac{1}{p_j}$ for any $i,j$ or equivalently $p_i = p_j$.

And hence $$p_i=\frac{1}{n}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.