Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am trying to calculate the relative entropy given two collections and have a question regarding some issues.

Supposed we have two sets, $Real$ and $Calculated$, and their respective probability mass functions, $P$, and $Q$.

Relative Entropy, or Kullback Leibler Divergence is defined as the following:

$$\sum_{i=0}^{n} P(i)\log \frac{P(i)}{Q(i)}$$

How do we properly handle situations where $|Q| \neq |P|$?

Should we take the intersection of the sets, $Real$ and $Calculated$, and scale their respective probability mass functions to correct the calculation of the relative entropy? Otherwise only calculating over the intersection without scaling the probabilities can lead to negative results, which is not correct.

I am using the following code to calculate R.E.

def kullback_leibler_divergence(real, predicted):

    sum = 0.0
    for qs, freq in predicted.items():
        freq_r = real.get(qs, 0.0)
        operand = m.log(freq_r / freq) if freq_r != 0 and freq != 0 else 0
        sum += freq_r * operand

    return sum

...but I get negative results occasionally, which made me question how I am handling my input parameters.

share|cite|improve this question
up vote 2 down vote accepted

The question is addressed here (see also @cardinal's answer on the linked page).

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.