Let us consider the following two probability distributions

P       Q
0.01    0.002
0.02    0.004
0.03    0.006
0.04    0.008
0.05    0.01
0.06    0.012
0.07    0.014
0.08    0.016
0.64    0.928

I have have calculated the Kullback-Leibler divergence which is equal $0.492820258$, I want to know in general what does this number shows me? In general Kullback divergence shows me how far is one probability distribution from another right? It is similar to entropy terminology, but in terms of number, how can I say let says result is 0.49 can i say that approximately one distribution is far from another by by 50%? or? thanks in advance


i have have calculated kullback-leibler divergence which is equal 0.492820258

  1. The KL divergence is not a dimensionless number, it has an unit (which depends on the base of the logarithm used), you must specify it unless it's implied by the context. I guess you used base 2, hence the unit it bits.

  2. The KL divergence (or "distance") is not symmetric: $D(p || q) \ne D(q|| p)$, so , again, you must specify which one you computed.

In our case $D(p||q)=0.49282$ bits.

Regarding the numerical significance, you could first compute the entropies. In our case $H(p)=1.9486...$ and $H(q)=0.5745...$ (always in bits). In terms of source encoding (first Shannon theorem), this says that source $p$ can be optimally encoded with $1.9486...$ bits per symbol. Now, if we encode source $p$ assumming (wrongly) that its true distribution were that of $q$, we'd get an average code length of $2.44142...$ bits per symbol (you can do the math). The "excess" error, the ineficciency that arises because of assuming a wrong distribution, is quantified by the KL divergence: $2.44142-1.9486=0.49282$ bits.

  • $\begingroup$ thanks for replay,yes i used base two and D(p||q),sorry i have confused it means that it shows us error of mistakenly assuming that some model is from one distribution,while in really it is from another one? $\endgroup$ – dato datuashvili Aug 11 '14 at 17:24
  • $\begingroup$ and also how you have computed 2.44142 ? $\endgroup$ – dato datuashvili Aug 11 '14 at 17:27
  • $\begingroup$ @datodatuashvili I suggest you to care a little more about your typing: uppercases, punctuation, spaces (eg, after commas). Not only as a cortesy, but also to make your text easier to read. $\endgroup$ – leonbloy Aug 11 '14 at 18:24
  • $\begingroup$ @datodatuashvili That is one of the possible interpretations of the KL distance, yes. It's mentioned in any textbook, or in the Wikipedia. $\endgroup$ – leonbloy Aug 11 '14 at 18:26
  • $\begingroup$ > how you have computed 2.44142? As $\sum_{x_i \in p} P(x_i) L_i $ where $L_i = -\log P^*(x_i)$ is the optimal length of the code for symbol $i$, compute from a probability function $P^*$ (in this case, from $q$) $\endgroup$ – leonbloy Aug 11 '14 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.