Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Can I determine the Kullback-Leibler distance $$ D_{\mathrm{KL}}(P\parallel Q)=\sum_{i}\ln\left(\frac{P(i)}{Q(i)}\right) P(i) $$ between the following probability distributions?

P(X) =  1   2   3   4  5   6    7   8         Q(X) =  1    2    3    4    5    6    7    8  
        2/8 1/8 1/8 0  1/8 2/8  0   1/8               0    1/8  1/8  2/8  1/8  0    2/8  1/8

These are 2 different probability distributions and sometimes in the sum $\ln(P(i)/Q(i)) = \infty$ and $P(i)=0$. Because somethimes the $P(i)$ or $Q(i)$ is $0$. How I can handle it?

share|cite|improve this question
You can't -- the KL divergence is only defined when the zeroes of $P$ and $Q$ are in the same places. – mjqxxxx Oct 23 '12 at 16:04
You might consider using the so-called "earth mover's distance" instead, unless you have a special reason for wanting KL in particular. – mjqxxxx Oct 23 '12 at 16:07
ahh i found this line in the definition of K-L: The K-L divergence is only defined if P and Q both sum to 1 and if implies for all i (absolute continuity). If the quantity appears in the formula, it is interpreted as zero because ." Please answer the question, and I accept as solution. – Tatar Elemér Oct 23 '12 at 17:19
up vote 0 down vote accepted

As the others already mentioned for the given data there exist no exact calculation of the KL divergence. However if you want to have something in hand anyway, then you can use deliver an $\epsilon$ amount of probability from the non zero probabilites to the zero probabilites, say $1/100$ from each. Then you wil have $3/100$ for the zero probabilities. Now you can calculate the KL-divergence.

In the next step deliver even less amount of probability say $1/1000$ and in the same way calculate the KL- divergence. This procedure will eventually give some ideas. It is also okay to remove a very tiny amount of probability from non zero elements because in the divisions or multiplications it will have a little effect while it will make your problem well defined.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.