enter image description here

This is the q(xi) distribution of information theory
(1/2) is bit information and $l_{i}$ is the length of the number of bit

then the cross enthropy derived to

enter image description here

I'm lack of experiecne to handling the log so can you show me how $-E_{p}[ln(q(x))/ln(2)]$ derived?

I want to know how

enter image description here


enter preformatted text here


1 Answer 1


Change of base formula in logarithm states, $$\log_ba \times \log_xb=\log_xa$$

Then $$-\mathbb{E}_P\frac{\ln q(X)}{\ln2}=-\mathbb{E}\log_2 q(X)$$

rest is writing out the expectation explicitly.

The context is unclear, however as $q(x_i)=\frac{1}{2^{l_i}}$. Taking $\ln$ we obtain, $$\ln q(x_i)=l_i \ln 2^{-1} \implies l_i=-\frac{\ln q(x_i)}{\ln2}$$. Taking expectation on both sides with respect to the disribution $P$ it follows.

  • $\begingroup$ I want to know the previous step. I edit the question $\endgroup$
    – slowmonk
    Jun 23, 2020 at 0:20
  • $\begingroup$ This hopefully answers your query $\endgroup$
    – Dinesh
    Jun 23, 2020 at 8:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .