I am not sure how to put the question. I am not even sure if this question makes sense at all.

I know that the similarity of two discrete (or continuous) distributions can be quantified by Kullback–Leibler distance. However, I wonder if it makes sense to quantify the Kullback–Leibler distance between two random variables which one is discrete and the other one is continuous?

Is there any probabilistic measure for quantifying the similarity of continuos distribution with a discrete one.

  • $\begingroup$ You could discretise the continuous distribution to have the same grid and then do the similarity calculation? Or you could fit/interpolate the discrete distribution to obtain a continuous distribution. All this is just ideas, and depends on the quality of the distributions. $\endgroup$ – Chinny84 May 27 '14 at 12:14

There exist a lot of statistical distances for two distributions. http://en.wikipedia.org/wiki/Statistical_distance What motivated you to use KL convergence? Although it is often intuited as a metric or distance, the KL divergence is not a true metric — for example, it is not symmetric: the KL divergence from P to Q is generally not the same as that from Q to P.

But if you want to extend the definition of the KL divergence to the case when $X$ is continuous and $Y$ is discrete I think I know how to do it.

Any discrete PMF (probability measure function) can be represented as a continuous PDF using a Dirac delta function. http://en.wikipedia.org/wiki/Dirac_delta_function

  • $\begingroup$ Thanks.I am struggling with an information theoretic problem where a set is discrete in a given domain. I do not know the exact cardinality of the set but I know the entries are lattice points (with different PMD) inside a k-ball with normalised second moment $M$. The origin $\boldsymbol{0}$ has the highest probability and the outermost points have lowest probability. Also I know the slope of reduction form centre to the outer shell. I am trying to model the discrete lattice inside the ball to an equivalent continuous layered spheres with the same slope and second moment. $\endgroup$ – M.X May 27 '14 at 16:51

In more details, the Dirac delta function $\delta(x)$ is defined as follows $$ \delta(x) = \begin{cases} \ +\infty, & x = 0 \\ \ 0, & x \neq 0 \end{cases} $$ and also $$ \int_{-\infty}^\infty \delta(x) = 1. $$ Now if your discrete PMD is defined as $P(k) = P(Y = k) = p_k$ for all integer $k$ then its PDF has the form $$ p_Y(y) = \sum_{k=-\infty}^{\infty} p_k \delta(y-k). $$ Indeed, for every infinitesimal $\epsilon > 0$ we have $$ P(k - \epsilon < Y < k + \epsilon) = \int_{k - \epsilon}^{k + \epsilon} p_k \delta(y-k) dy = p_k. $$ Further, $$ D_{KL}(X||Y) = \int_{-\infty}^\infty \ln(\frac{p_X(x)}{p_Y(x)})p_X(x) dx = \ldots $$

Probably, this approach will give some result...


You can't do KL between discrete and continuous usefully.

If you did KL between discrete and continuous, you'd have to do D(discrete||continuous) (else, it would always be infinite due to supports). And even then, theres no mass at a single point in a continuous distribution, so it doesn't really have compatibility there either (if you wanted the full version of KL divergence, you need to have $D(p||q)$ to be such that $p << \mu, q << \mu$ and then define this in terms of their Radon-Nikodym derivatives).

See this thread as well.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.