0
$\begingroup$

the proof of the concavity of entropy

In the proof above it is hard for me to comprehend the equation, especially the $\log|X|$. What is the $u$, the so-called "the uniform distribution on $|X|$ outcomes"?

Any kind of suggestions for me to understand the equation or the theorem are highly appreciated.

$\endgroup$
3
$\begingroup$

$$u(x)=\frac{1}{|X|}, \forall x \in X$$

\begin{align}H(p)&=-\sum_{x \in X} p(x) \log p(x) \\& = -\sum_{x \in X} p(x) \log u(x)-\sum_{x \in X} p(x) \log p(x)+\sum_{x \in X} p(x) \log u(x) \\ & =-\sum_{x \in X} p(x) \log u(x) - D(p ||u) \\ &=-\sum_{x \in X} p(x) \log \frac{1}{|X|} - D(p ||u) \\ \\ & = - \log \frac{1}{|X|} \sum_{x \in X} p(x)- D(p ||u) \\ \\ & = \log(|X|) \sum_{x \in X} p(x)- D(p ||u) \\ \\ & = \log(|X|) - D(p ||u) \\ \end{align}

$\endgroup$
  • $\begingroup$ Thanks very much for your detailed mathematical speculation. But could you please teach me any thing about the form of $\log(|X|)$? What is the relation between the cross entropy and that? $\endgroup$ – Lerner Zhang Nov 5 '16 at 7:43
  • $\begingroup$ I know that it is for the proof, but any thing else, considering the saying of "uniform distribution on $|X|$ outcomes" Thanks. $\endgroup$ – Lerner Zhang Nov 5 '16 at 7:44
  • 1
    $\begingroup$ $\log (|X|)$ is a constant isn't it? $\endgroup$ – Siong Thye Goh Nov 5 '16 at 7:52
  • 1
    $\begingroup$ Let $X = \left\{1,2,3 \right\}$, $u_X(1)=u_X(2)=u_X(3)=\frac13$ which is a constant and $\log (|X|) =\log 3$. $\endgroup$ – Siong Thye Goh Nov 5 '16 at 8:00
  • 1
    $\begingroup$ $|X|$ is just the cardinality of the alphabet. In example of coin tosses where you just care about head or tail, $|X|=2$. If you are interested in the outcome of a dice regardless of whether it is biased or not, $|X|=6$. $\endgroup$ – Siong Thye Goh Nov 5 '16 at 8:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.