I have just come across the Kullback-Leibler divergence in an article. This is the first time I am seeing it. What intrigued me are the two lines '||' in $ D_{KL}(P‖Q)$. Below is the formulae. enter image description here

I am not sure what they mean. Do they mean 'with respect to'?

Can anyone kindly clarfiy this for me?

Thanking you in advance.

  • 2
    $\begingroup$ They don't mean anything really, it's just a way to separate $P$ and $Q$. $\endgroup$ Nov 24, 2017 at 4:43
  • $\begingroup$ That's it? :P Really hoped it would signify something else. $\endgroup$
    – nashynash
    Nov 24, 2017 at 4:51
  • $\begingroup$ And to show it is non-commutative in $P,Q$ (in the way as $p(x|z)\ $ ) $\endgroup$
    – reuns
    Nov 24, 2017 at 5:01
  • $\begingroup$ @reuns I am sorry I did not get that. Could you please elaborate on that point? $\endgroup$
    – nashynash
    Nov 24, 2017 at 5:04
  • $\begingroup$ $D_{KL}(X\| Y) = \mathbb{E}_X[-\log \frac{p(X= x)}{p(Y=x)}]$. $\endgroup$
    – reuns
    Nov 24, 2017 at 5:22


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.