Well known formula of KL divergence when we have a discrete probability distributions.

$$D_{KL}(P \parallel Q)=\sum\limits_i \ln \left(\frac{P(i)}{Q(i)}\right) P(i)$$

Can someone explain why the natural base of the logarithm? That will probably not yield the information in bits as a result?

Do I need to change the base of the logarithm to 2 in order to get the relative entropy in bits?

Or there is another way?

Thank you.



$$ \log_2 x = \frac{\log_e x}{\log_e 2} = \frac{\ln x}{\ln 2} $$

Expressing entropy in bits means using base-$2$ logarithms. Just divide the base-$e$ logarithms by $\ln 2$ and you've got it.

  • $\begingroup$ Thanks Michael, Can you tell me is this correct way of expressing KLD? $D_{KL}(P \parallel Q)=\sum\limits_{i} \log _2 (\frac{P(i)}{Q(i)}) P(i)$ $\endgroup$ – user2703038 Nov 23 '13 at 21:33
  • $\begingroup$ Yes. ${{{{{{{}}}}}}}$ $\endgroup$ – Michael Hardy Nov 23 '13 at 21:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.