# Definitions of K-L divergence based on likelihood ratio and on R-N derivative

From Wikipedia

For distributions $P$ and $Q$ of a continuous random variable, KL-divergence is defined to be the integral: $$D_{\mathrm{KL}}(P\|Q) = \int_{-\infty}^\infty \ln\left(\frac{p(x)}{q(x)}\right) p(x) \, {\rm d}x, \!$$ where $p$ and $q$ denote the densities of $P$ and $Q$.

More generally, if $P$ and $Q$ are probability measures over a set $X$, and $P$ is absolutely continuous with respect to $Q$, then the Kullback–Leibler divergence from $P$ to $Q$ is defined as $$D_{\mathrm{KL}}(P\|Q) = \int_X \ln\left(\frac{{\rm d}P}{{\rm d}Q}\right) \,{\rm d}P, \!$$ where $\frac{{\rm d}P}{{\rm d}Q}$ is the Radon–Nikodym derivative of $P$ with respect to $Q$, and provided the expression on the right-hand side exists.

If I understand correctly, R-N derivative is same as likelihood ratio, but the latter can still exist when the former doesn't exist, i.e. when one measure isn't absolute continuous wrt the other measure.

The first definition of K-L divergence is based on likelihood ratio, and the second definition is based on R-N derivative. So how is the second "more general"?

Thanks and regards!

-

The first definition only applies to probability measures on $\mathbb{R}$ which have densities. Not all probability measures are defined on $\mathbb{R}$, and not all probability measures defined on $\mathbb{R}$ have densities.
Thanks, that makes sense. In the first definition, $P$ may not be absolutely continuous to $Q$, isn't it? So the second definition doesn't generalize the first one? – Tim May 1 '13 at 21:06