2
$\begingroup$

The Kullback-Leibler (KL) divergence (which I am familiar with) is usually defined for probability distributions. However, even though some people claim that a probability distribution is just a synonym for probability measure, that is, some people claim that a probability distribution is a function $\mathbb{P}$ from the event space $\mathcal{F}$ to $[0, 1]$, or, more formally, $\mathbb{P}: \mathcal{F} \rightarrow [0, 1]$, other people claim that the definition of a probability distribution depends on the context.

For example, the Wikipedia article on the KL divergence defines the KL divergence as follows

For discrete probability distributions $P$ and $Q$ defined on the same probability space, $\mathcal {X}$, the Kullback–Leibler divergence of $Q$ from $P$ is defined to be

$$ D_\text{KL}(P \parallel Q) = \sum_{x\in\mathcal{X}} P(x) \log\left(\frac{P(x)}{Q(x)}\right) $$

A probability space $\mathcal{X}$ is usually defined as a triple consisting of the sample space $\Omega$, the event space $\mathcal{F}$ and the probability measure $\mathbb{P}$. So, it seems that the definition above says that a probability distribution (or probability measure, assuming that a probability distribution is a synonym for probability measure) is defined on a probability space, which is composed of a probability measure, so it seems that it states that probability distribution is defined on a probability distribution, which makes little or no sense. So, in the definition above, $P$ and $Q$ should not be probability measures. What's going on?

Is the Kullback-Leibler divergence defined for probability measures or random variables, or something else (e.g. cumulative distribution functions)?

$\endgroup$
1
  • $\begingroup$ I think they are just saying "$P$ and $Q$ are defined on the same measurable space $(\Omega, \mathcal{F})$." $\endgroup$
    – angryavian
    Commented Dec 21, 2019 at 19:48

1 Answer 1

2
$\begingroup$

For two probability measures $\nu$ and $\mu$ s.t. $\nu\ll\mu$, the KL divergence of $\nu$ w.r.t. $\mu$ is defined as $$ D_{KL}(\nu\,||\,\mu)=\mathsf{E}^{\nu}\ln\left(\frac{d\nu}{d\mu}\right),\label{1}\tag{1} $$ where $d\nu/d\mu$ is the Radon-Nikodym derivative of $\nu$ w.r.t. $\mu$. If $\nu$ is not absolutely continuous w.r.t. $\mu$, $D_{KL}(\nu\,||\,\mu)=\infty$.


Indeed, a probability distribution is a probability measure (typically, on $\mathbb{R}^d$). However, when random variables are involved, their distributions are push-forward measures. In this case $D_{KL}$ "between random variables" is $\eqref{1}$ applied to the corresponding distribution functions.

$\endgroup$
2
  • $\begingroup$ Your answer may be correct, but it doesn't help me much, because you don't try to relate it to the definition of KL divergence in my post. According to this answer, the KL divergence is defined for probability measures (aka probability distributions), but your definition doesn't look very similar to the one in my post. How does your definition translate to the cases where we have discrete random variables? I would also appreciate if you could use a notation similar to the one used in the Wikipedia article. $\endgroup$
    – user168764
    Commented Dec 21, 2019 at 22:30
  • $\begingroup$ Your definition is a particular case of (1). You may easily work out the case of discrete random variables; just note that the Radon-Nikodym derivative in this case is the ratio of the corresponding probability mass functions. By the way, look at this wiki article. $\endgroup$
    – user140541
    Commented Dec 22, 2019 at 9:11

You must log in to answer this question.