The Kullback-Leibler divergence between two (discrete) probability distributions is defined as $$ D_{KL}(P\|Q) = \sum_i p_i \log \frac{p_i}{q_i}, $$ where $p_i$ is the probability that $P$ assigns to the event $i$, and $q_i$ is the probability assigned by $Q$.

I know that the quantity $D_{KL}(P\|Q) + D_{KL}(Q\|P)$ (symmetrised Kullback-Leibler divergence) is sometimes used, because it is symmetric and thus behaves more like a distance between the two distributions. But does anyone know of a case where the quantity $$ \sum_i (p_i-q_i) \log \frac{p_i}{q_i} $$ is used, and whether it has a standard name? I ask because it came up in some statistical mechanics work I'm doing and I want to know if it has an interpretation in terms of information theory, or any particularly interesting known properties.

  • 2
    $\begingroup$ Don't we have $(p - q)\log \frac pq = p \log\frac pq + q \log \left(\frac pq\right)^{-1} = p\log\frac pq + q \log \frac qp$, so the quantity you are looking at is the symmetrised KL-divergence? $\endgroup$ – martini Mar 19 '12 at 11:20
  • $\begingroup$ D'oh - yes, you're right. I kept getting confused with the signs and at first thought it was $D_{KL}(P\|Q)-D_{KL}(Q\|P)$ (hence mentioning the symmetrised KL-divergence in the question). Then I realised that wasn't right, but I didn't spot that it's just equal to the symmetrised KL-divergence. So this is actually a pretty silly question - not sure whether I should just delete it. $\endgroup$ – Nathaniel Mar 19 '12 at 11:29

Since \begin{align*} \sum_i (p_i - q_i) \log \frac{p_i}{q_i} &= \sum_i p_i \log \frac{p_i}{q_i} - \sum_i q_i \log \frac{p_i}{q_i}\\ &= \sum_i p_i \log \frac{p_i}{q_i} + \sum_i q_i \log \left(\frac{p_i}{q_i}\right)^{-1}\\ &= D_{KL}(P||Q) + D_{KL}(Q||P) \end{align*} the quantity in question is the symmetrised KL-divercence,


  • $\begingroup$ how to reconcile the fact that this symmetrised KL-divergence gives a solution that is virtually twice as large as any of its individual components, $D_{KL}(P||Q)$ and $D_{KL}(Q||P)$? $\endgroup$ – develarist Nov 9 '20 at 8:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.