0
$\begingroup$

KL divergence is generally written for continuous or discrete densities. I am interested in the case where we have a joint density with both continuous and discrete variables. If KL divergence can be defined for this case, what does it look like? What assumptions do we need?

As an example, consider two joint densities $p(x,y)$ and $q(x,y)$, where $x\sim\operatorname{Norm}(\mu,\Sigma)$ and $y\sim\operatorname{Cat}(\rho)$. In this case, $\operatorname{Cat}(\rho)$ is the categorical distribution as described here. What would $\operatorname{KL}(p||q)$ look like? It is okay to assume that $x$ and $y$ are independent, or conditionally independent given their parameters, but I'd like to know if this is a necessary assumption.

$\endgroup$

1 Answer 1

1
$\begingroup$

If $p,\,q$ are discrete distributions with the same support $\{x_i|i\in I\}$, they can be regarded as continuous using Dirac deltas. In particular, the PDF of the first distribution is $p(x)=\sum_ip_i\delta(x-x_i)$ for probability masses $p_i$. Treating $q$ similarly,$$\int_\mathbb{R}p(x)\ln\frac{p(x)}{q(x)}dx=\sum_i\int_\mathbb{R}p_i\ln\frac{p_i}{q_i}dx.$$In other words, the continuous definition of $D_\text{KL}$ (Eq. 2 here) becomes the discrete case (Eq. 1, ibid.) in the secretly-really-continuous case. So if in the mixed case we can use Dirac deltas to write a "probability density" (it will be a measure rather than a true function), we can try evaluating the integral as usual. Note, however, it might not exist, depending on where the distributions have nonzero densities.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .