4
$\begingroup$

The Kullback-Leibler divergence between two (discrete) probability distributions is defined as $$ D_{KL}(P\|Q) = \sum_i p_i \log \frac{p_i}{q_i}, $$ where $p_i$ is the probability that $P$ assigns to the event $i$, and $q_i$ is the probability assigned by $Q$.

I know that the quantity $D_{KL}(P\|Q) + D_{KL}(Q\|P)$ (symmetrised Kullback-Leibler divergence) is sometimes used, because it is symmetric and thus behaves more like a distance between the two distributions. But does anyone know of a case where the quantity $$ \sum_i (p_i-q_i) \log \frac{p_i}{q_i} $$ is used, and whether it has a standard name? I ask because it came up in some statistical mechanics work I'm doing and I want to know if it has an interpretation in terms of information theory, or any particularly interesting known properties.

$\endgroup$
2
  • 2
    $\begingroup$ Don't we have $(p - q)\log \frac pq = p \log\frac pq + q \log \left(\frac pq\right)^{-1} = p\log\frac pq + q \log \frac qp$, so the quantity you are looking at is the symmetrised KL-divergence? $\endgroup$
    – martini
    Commented Mar 19, 2012 at 11:20
  • $\begingroup$ D'oh - yes, you're right. I kept getting confused with the signs and at first thought it was $D_{KL}(P\|Q)-D_{KL}(Q\|P)$ (hence mentioning the symmetrised KL-divergence in the question). Then I realised that wasn't right, but I didn't spot that it's just equal to the symmetrised KL-divergence. So this is actually a pretty silly question - not sure whether I should just delete it. $\endgroup$
    – N. Virgo
    Commented Mar 19, 2012 at 11:29

1 Answer 1

6
$\begingroup$

Since \begin{align*} \sum_i (p_i - q_i) \log \frac{p_i}{q_i} &= \sum_i p_i \log \frac{p_i}{q_i} - \sum_i q_i \log \frac{p_i}{q_i}\\ &= \sum_i p_i \log \frac{p_i}{q_i} + \sum_i q_i \log \left(\frac{p_i}{q_i}\right)^{-1}\\ &= D_{KL}(P||Q) + D_{KL}(Q||P) \end{align*} the quantity in question is the symmetrised KL-divercence,

AB,

$\endgroup$
1
  • $\begingroup$ how to reconcile the fact that this symmetrised KL-divergence gives a solution that is virtually twice as large as any of its individual components, $D_{KL}(P||Q)$ and $D_{KL}(Q||P)$? $\endgroup$
    – develarist
    Commented Nov 9, 2020 at 8:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .