Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

The Kullback-Leibler divergence between two (discrete) probability distributions is defined as $$ D_{KL}(P\|Q) = \sum_i p_i \log \frac{p_i}{q_i}, $$ where $p_i$ is the probability that $P$ assigns to the event $i$, and $q_i$ is the probability assigned by $Q$.

I know that the quantity $D_{KL}(P\|Q) + D_{KL}(Q\|P)$ (symmetrised Kullback-Leibler divergence) is sometimes used, because it is symmetric and thus behaves more like a distance between the two distributions. But does anyone know of a case where the quantity $$ \sum_i (p_i-q_i) \log \frac{p_i}{q_i} $$ is used, and whether it has a standard name? I ask because it came up in some statistical mechanics work I'm doing and I want to know if it has an interpretation in terms of information theory, or any particularly interesting known properties.

share|cite|improve this question
Don't we have $(p - q)\log \frac pq = p \log\frac pq + q \log \left(\frac pq\right)^{-1} = p\log\frac pq + q \log \frac qp$, so the quantity you are looking at is the symmetrised KL-divergence? – martini Mar 19 '12 at 11:20
D'oh - yes, you're right. I kept getting confused with the signs and at first thought it was $D_{KL}(P\|Q)-D_{KL}(Q\|P)$ (hence mentioning the symmetrised KL-divergence in the question). Then I realised that wasn't right, but I didn't spot that it's just equal to the symmetrised KL-divergence. So this is actually a pretty silly question - not sure whether I should just delete it. – Nathaniel Mar 19 '12 at 11:29
I'll post an answer, so if you choose to keep, you can accept. – martini Mar 19 '12 at 12:01
up vote 3 down vote accepted

Since \begin{align*} \sum_i (p_i - q_i) \log \frac{p_i}{q_i} &= \sum_i p_i \log \frac{p_i}{q_i} - \sum_i q_i \log \frac{p_i}{q_i}\\ &= \sum_i p_i \log \frac{p_i}{q_i} + \sum_i q_i \log \left(\frac{p_i}{q_i}\right)^{-1}\\ &= D_{KL}(P||Q) + D_{KL}(Q||P) \end{align*} the quantity in question is the symmetrised KL-divercence,


share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.