Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am trying to find the Kullback–Leibler divergence between Bernoulli Distribution on two points $T, -T$ with parameter $p$ and Gaussian Distribution with mean $\mu$ and variance $\sigma^2$. My attempt is as follows:

Let $$ b(x) = q\delta(x-T)+p\delta(x+T) \sim \text{Bernoulli}(p) \\ g(x) \sim N(\mu, \sigma^2). $$

$$ \begin{align} D(b||g) &= \int_{-\infty}^{\infty}b(x)\log \left( \frac{b(x)}{g(x)}\right) dx \\ &=\int_{-\infty}^{\infty}b(x)\log \left( b(x) \right) dx - \int_{-\infty}^{\infty}b(x)\log \left( g(x) \right) dx \\ &=A-B \end{align} $$

My questions are as follows:

  1. Can I use the continuous representation of Bernoulli RV with the help of $\delta(.)$ functions where $\delta(.)$ is Dirac Delta function?
  2. Does $A$ exist? Because, on the set $\mathbb{R}-\{\pm T\}$, $\log(\delta(x \mp T))$ is $-\infty$.
  3. If we cannot calculate the KLD between a continuous and a discrete random variable, what is the KLD analogue for this case? My thought was that $B$ alone can serve as a distance. For example, if we want to measure the distance of $b(x)$ from two different Gaussian distributions $g_1(x), g_2(x)$, only $B$ depends on $g_1(x)$ or $g_2(x)$, and thus can contribute to KLD.
share|cite|improve this question
up vote 1 down vote accepted
  1. No, you cannot do this. The Kullback-Leibler divergence $D_{KL}(P\|Q)$ is defined only if $P\ll Q$. This means that no set of positive $P$-measure can have zero $Q$-measure. In your case it will not work because the point masses of the Bernoulli distribution have zero measure under the Gaussian distribution.
  2. The integral A blows up. For a continuous distribution this would the negative of the continuous version of the Shannon entropy.
  3. I suspect that you might be looking for the mutual information between parameter space and observation space. It is a common technique to try to maximize the mutual information in such settings. The mutual information is then equal to the expected Kullback-Leibler divergence of the posterior distribution on a parameter space (given the observations) from the prior distribution. Here, the requirement that $P\ll Q$ simply means that one is not allowed to make any conclusions that are a priori impossible!
share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.