17
$\begingroup$

The Kullback-Leibler divergence has a strong relationship with mutual information, and mutual information has a number of normalized variants. Is there some similar, entropy-like value that I can use to normalize KL-divergence such that the normalized KL-divergence is bounded above by 1 (and below by 0)?

$\endgroup$
6
  • $\begingroup$ $1-\exp( -D_{KL} )$ suggests itself. $\endgroup$
    – Emre
    Jul 14, 2011 at 19:09
  • $\begingroup$ @Emre I was hoping for something more along the lines of most of those MI examples where the measure is simply divided by some entropy-like calculation. $\endgroup$ Jul 14, 2011 at 22:00
  • $\begingroup$ To do that you would need to know the distribution maximally divergent from the reference. $\endgroup$
    – Emre
    Jul 14, 2011 at 22:14
  • $\begingroup$ ...and the divergence is unbounded if one distribution has zero density where the other does not (think of the logarithm), so this seems impossible without further constraints. $\endgroup$
    – Emre
    Jul 15, 2011 at 17:02
  • $\begingroup$ @Emre I think you've convinced me that it's likely impossible in the general case. It so happens I was primarily interested in cases where the two distributions share a particular relationship with one another, and with that I was able to find a bound (the situation you described cannot happen with my classes of distributions). I think your comments describing why this can't be done in the general case are the correct answer for the question as written, so if you want to write them into an answer then I can accept. $\endgroup$ Jul 15, 2011 at 18:57

2 Answers 2

8
$\begingroup$

In the most general class of distributions your multiplicative normalization approach is not possible because one can trivially select the comparison density to be zero in some interval leading to an unbounded divergence. Therefore your approach only makes sense with positive densities, or where both share the same support (perhaps densities of the same class?)

Instead you might consider normalization through a nonlinear transformation such as $1-\exp(-D_{KL})$.

$\endgroup$
2
$\begingroup$

The KL is only defined for distributions having compatible supports... Nevertheless, it is true that one can find distributions that have infinite KL divergence.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .