# Is the Kullback-Leibler divergence defined for probability measures or random variables?

The Kullback-Leibler (KL) divergence (which I am familiar with) is usually defined for probability distributions. However, even though some people claim that a probability distribution is just a synonym for probability measure, that is, some people claim that a probability distribution is a function $$\mathbb{P}$$ from the event space $$\mathcal{F}$$ to $$[0, 1]$$, or, more formally, $$\mathbb{P}: \mathcal{F} \rightarrow [0, 1]$$, other people claim that the definition of a probability distribution depends on the context.

For example, the Wikipedia article on the KL divergence defines the KL divergence as follows

For discrete probability distributions $$P$$ and $$Q$$ defined on the same probability space, $$\mathcal {X}$$, the Kullback–Leibler divergence of $$Q$$ from $$P$$ is defined to be

$$D_\text{KL}(P \parallel Q) = \sum_{x\in\mathcal{X}} P(x) \log\left(\frac{P(x)}{Q(x)}\right)$$

A probability space $$\mathcal{X}$$ is usually defined as a triple consisting of the sample space $$\Omega$$, the event space $$\mathcal{F}$$ and the probability measure $$\mathbb{P}$$. So, it seems that the definition above says that a probability distribution (or probability measure, assuming that a probability distribution is a synonym for probability measure) is defined on a probability space, which is composed of a probability measure, so it seems that it states that probability distribution is defined on a probability distribution, which makes little or no sense. So, in the definition above, $$P$$ and $$Q$$ should not be probability measures. What's going on?

Is the Kullback-Leibler divergence defined for probability measures or random variables, or something else (e.g. cumulative distribution functions)?

• I think they are just saying "$P$ and $Q$ are defined on the same measurable space $(\Omega, \mathcal{F})$." Commented Dec 21, 2019 at 19:48

For two probability measures $$\nu$$ and $$\mu$$ s.t. $$\nu\ll\mu$$, the KL divergence of $$\nu$$ w.r.t. $$\mu$$ is defined as $$D_{KL}(\nu\,||\,\mu)=\mathsf{E}^{\nu}\ln\left(\frac{d\nu}{d\mu}\right),\label{1}\tag{1}$$ where $$d\nu/d\mu$$ is the Radon-Nikodym derivative of $$\nu$$ w.r.t. $$\mu$$. If $$\nu$$ is not absolutely continuous w.r.t. $$\mu$$, $$D_{KL}(\nu\,||\,\mu)=\infty$$.
Indeed, a probability distribution is a probability measure (typically, on $$\mathbb{R}^d$$). However, when random variables are involved, their distributions are push-forward measures. In this case $$D_{KL}$$ "between random variables" is $$\eqref{1}$$ applied to the corresponding distribution functions.