Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I am trying to calculate the relative entropy given two collections and have a question regarding some issues.

Supposed we have two sets, $Real$ and $Calculated$, and their respective probability mass functions, $P$, and $Q$.

Relative Entropy, or Kullback Leibler Divergence is defined as the following:

$$\sum_{i=0}^{n} P(i)\log \frac{P(i)}{Q(i)}$$

How do we properly handle situations where $|Q| \neq |P|$?

Should we take the intersection of the sets, $Real$ and $Calculated$, and scale their respective probability mass functions to correct the calculation of the relative entropy? Otherwise only calculating over the intersection without scaling the probabilities can lead to negative results, which is not correct.

I am using the following code to calculate R.E.

def kullback_leibler_divergence(real, predicted):

    sum = 0.0
    for qs, freq in predicted.items():
        freq_r = real.get(qs, 0.0)
        operand = m.log(freq_r / freq) if freq_r != 0 and freq != 0 else 0
        sum += freq_r * operand

    return sum

...but I get negative results occasionally, which made me question how I am handling my input parameters.

share|improve this question
add comment

1 Answer 1

up vote 2 down vote accepted

The question is addressed here (see also @cardinal's answer on the linked page).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.