Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

From Wikipedia

For distributions $P$ and $Q$ of a continuous random variable, KL-divergence is defined to be the integral: $$ D_{\mathrm{KL}}(P\|Q) = \int_{-\infty}^\infty \ln\left(\frac{p(x)}{q(x)}\right) p(x) \, {\rm d}x, \! $$ where $p$ and $q$ denote the densities of $P$ and $Q$.

More generally, if $P$ and $Q$ are probability measures over a set $X$, and $P$ is absolutely continuous with respect to $Q$, then the Kullback–Leibler divergence from $P$ to $Q$ is defined as $$ D_{\mathrm{KL}}(P\|Q) = \int_X \ln\left(\frac{{\rm d}P}{{\rm d}Q}\right) \,{\rm d}P, \! $$ where $\frac{{\rm d}P}{{\rm d}Q}$ is the Radon–Nikodym derivative of $P$ with respect to $Q$, and provided the expression on the right-hand side exists.

If I understand correctly, R-N derivative is same as likelihood ratio, but the latter can still exist when the former doesn't exist, i.e. when one measure isn't absolute continuous wrt the other measure.

The first definition of K-L divergence is based on likelihood ratio, and the second definition is based on R-N derivative. So how is the second "more general"?

Thanks and regards!

share|cite|improve this question

The first definition only applies to probability measures on $\mathbb{R}$ which have densities. Not all probability measures are defined on $\mathbb{R}$, and not all probability measures defined on $\mathbb{R}$ have densities.

share|cite|improve this answer
Thanks, that makes sense. In the first definition, $P$ may not be absolutely continuous to $Q$, isn't it? So the second definition doesn't generalize the first one? – Tim May 1 '13 at 21:06
I suppose that strictly speaking neither of them generalizes the other as written. – Qiaochu Yuan May 1 '13 at 21:58
Thanks, do you know a definition that can generalize both? – Tim May 1 '13 at 22:06

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.