The Kullback-Leibler divergence has a strong relationship with mutual information, and mutual information has a number of normalized variants. Is there some similar, entropy-like value that I can use to normalize KL-divergence such that the normalized KL-divergence is bounded above by 1 (and below by 0)?

  • $\begingroup$ $1-\exp( -D_{KL} )$ suggests itself. $\endgroup$ – Emre Jul 14 '11 at 19:09
  • $\begingroup$ @Emre I was hoping for something more along the lines of most of those MI examples where the measure is simply divided by some entropy-like calculation. $\endgroup$ – Michael McGowan Jul 14 '11 at 22:00
  • $\begingroup$ To do that you would need to know the distribution maximally divergent from the reference. $\endgroup$ – Emre Jul 14 '11 at 22:14
  • $\begingroup$ ...and the divergence is unbounded if one distribution has zero density where the other does not (think of the logarithm), so this seems impossible without further constraints. $\endgroup$ – Emre Jul 15 '11 at 17:02
  • $\begingroup$ @Emre I think you've convinced me that it's likely impossible in the general case. It so happens I was primarily interested in cases where the two distributions share a particular relationship with one another, and with that I was able to find a bound (the situation you described cannot happen with my classes of distributions). I think your comments describing why this can't be done in the general case are the correct answer for the question as written, so if you want to write them into an answer then I can accept. $\endgroup$ – Michael McGowan Jul 15 '11 at 18:57

In the most general class of distributions your multiplicative normalization approach is not possible because one can trivially select the comparison density to be zero in some interval leading to an unbounded divergence. Therefore your approach only makes sense with positive densities, or where both share the same support (perhaps densities of the same class?)

Instead you might consider normalization through a nonlinear transformation such as $1-\exp(-D_{KL})$.


The KL is only defined for distributions having compatible supports... Nevertheless, it is true that one can find distributions that have infinite KL divergence.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.