Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It only takes a minute to sign up.
Sign up to join this community
I am not sure how to put the question. I am not even sure if this question makes sense at all.
I know that the similarity of two discrete (or continuous) distributions can be quantified by Kullback–Leibler distance. However, I wonder if it makes sense to quantify the Kullback–Leibler distance between two random variables which one is discrete and the other one is continuous?
Is there any probabilistic measure for quantifying the similarity of continuos distribution with a discrete one.
There exist a lot of statistical distances for two distributions. http://en.wikipedia.org/wiki/Statistical_distance What motivated you to use KL convergence? Although it is often intuited as a metric or distance, the KL divergence is not a true metric — for example, it is not symmetric: the KL divergence from P to Q is generally not the same as that from Q to P.
But if you want to extend the definition of the KL divergence to the case when $X$ is continuous and $Y$ is discrete I think I know how to do it.
Any discrete PMF (probability measure function) can be represented as a continuous PDF using a Dirac delta function. http://en.wikipedia.org/wiki/Dirac_delta_function
In more details, the Dirac delta function $\delta(x)$ is defined as follows $$ \delta(x) = \begin{cases} \ +\infty, & x = 0 \\ \ 0, & x \neq 0 \end{cases} $$ and also $$ \int_{-\infty}^\infty \delta(x) = 1. $$ Now if your discrete PMD is defined as $P(k) = P(Y = k) = p_k$ for all integer $k$ then its PDF has the form $$ p_Y(y) = \sum_{k=-\infty}^{\infty} p_k \delta(y-k). $$ Indeed, for every infinitesimal $\epsilon > 0$ we have $$ P(k - \epsilon < Y < k + \epsilon) = \int_{k - \epsilon}^{k + \epsilon} p_k \delta(y-k) dy = p_k. $$ Further, $$ D_{KL}(X||Y) = \int_{-\infty}^\infty \ln(\frac{p_X(x)}{p_Y(x)})p_X(x) dx = \ldots $$
Probably, this approach will give some result...
You can't do KL between discrete and continuous usefully.
If you did KL between discrete and continuous, you'd have to do D(discrete||continuous) (else, it would always be infinite due to supports). And even then, theres no mass at a single point in a continuous distribution, so it doesn't really have compatibility there either (if you wanted the full version of KL divergence, you need to have $D(p||q)$ to be such that $p << \mu, q << \mu$ and then define this in terms of their Radon-Nikodym derivatives).
See this thread as well.
