Can I determine the Kullback-Leibler distance $$ D_{\mathrm{KL}}(P\parallel Q)=\sum_{i}\ln\left(\frac{P(i)}{Q(i)}\right) P(i) $$ between the following probability distributions?

P(X) =  1   2   3   4  5   6    7   8         Q(X) =  1    2    3    4    5    6    7    8  
        2/8 1/8 1/8 0  1/8 2/8  0   1/8               0    1/8  1/8  2/8  1/8  0    2/8  1/8

These are 2 different probability distributions and sometimes in the sum $\ln(P(i)/Q(i)) = \infty$ and $P(i)=0$. Because somethimes the $P(i)$ or $Q(i)$ is $0$. How I can handle it?

  • $\begingroup$ You can't -- the KL divergence is only defined when the zeroes of $P$ and $Q$ are in the same places. $\endgroup$ – mjqxxxx Oct 23 '12 at 16:04
  • 2
    $\begingroup$ You might consider using the so-called "earth mover's distance" instead, unless you have a special reason for wanting KL in particular. $\endgroup$ – mjqxxxx Oct 23 '12 at 16:07
  • $\begingroup$ ahh i found this line in the definition of K-L: The K-L divergence is only defined if P and Q both sum to 1 and if implies for all i (absolute continuity). If the quantity appears in the formula, it is interpreted as zero because ." Please answer the question, and I accept as solution. $\endgroup$ – Tatar Elemér Oct 23 '12 at 17:19
  • $\begingroup$ its $ = \infty$ $\endgroup$ – kjetil b halvorsen Jul 11 '17 at 10:32

As the others already mentioned for the given data there exist no exact calculation of the KL divergence. However if you want to have something in hand anyway, then you can use deliver an $\epsilon$ amount of probability from the non zero probabilites to the zero probabilites, say $1/100$ from each. Then you wil have $3/100$ for the zero probabilities. Now you can calculate the KL-divergence.

In the next step deliver even less amount of probability say $1/1000$ and in the same way calculate the KL- divergence. This procedure will eventually give some ideas. It is also okay to remove a very tiny amount of probability from non zero elements because in the divisions or multiplications it will have a little effect while it will make your problem well defined.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.