Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

The Kullback-Leibler divergence has a strong relationship with mutual information, and mutual information has a number of normalized variants. Is there some similar, entropy-like value that I can use to normalize KL-divergence such that the normalized KL-divergence is bounded above by 1 (and below by 0)?

share|cite|improve this question
$1-\exp( -D_{KL} )$ suggests itself. – Emre Jul 14 '11 at 19:09
@Emre I was hoping for something more along the lines of most of those MI examples where the measure is simply divided by some entropy-like calculation. – Michael McGowan Jul 14 '11 at 22:00
To do that you would need to know the distribution maximally divergent from the reference. – Emre Jul 14 '11 at 22:14
...and the divergence is unbounded if one distribution has zero density where the other does not (think of the logarithm), so this seems impossible without further constraints. – Emre Jul 15 '11 at 17:02
@Emre I think you've convinced me that it's likely impossible in the general case. It so happens I was primarily interested in cases where the two distributions share a particular relationship with one another, and with that I was able to find a bound (the situation you described cannot happen with my classes of distributions). I think your comments describing why this can't be done in the general case are the correct answer for the question as written, so if you want to write them into an answer then I can accept. – Michael McGowan Jul 15 '11 at 18:57
up vote 1 down vote accepted

In the most general class of distributions your multiplicative normalization approach is not possible because one can trivially select the comparison density to be zero in some interval leading to an unbounded divergence. Therefore your approach only makes sense with positive densities, or where both share the same support (perhaps densities of the same class?)

Instead you might consider normalization through a nonlinear transformation such as $1-\exp(-D_{KL})$.

share|cite|improve this answer

The KL is only defined for distributions having compatible supports... Nevertheless, it is true that one can find distributions that have infinite KL divergence.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.