I have two samples of probability distributions that I would like to compare. I have previously heard about the Kullback-Leibler divergence, but reading up on this it seems like its non-symmetricity makes it more suitable for comparing a sample to a model, rather than comparing two samples. What would you propose that I use instead, or maybe the KL divergence actually is a good choice?
$\begingroup$
$\endgroup$
2
-
2$\begingroup$ You can use the symmetric KL divergence, $D_{KL}(P, Q) + D_{KL}(Q, P)$. See here for more options: en.wikipedia.org/wiki/… . There are also many options that aren't based on the KL divergence, like statistical distance $\frac{1}{2}\int |P-Q|$, which is a metric. Even more options are here: en.wikipedia.org/wiki/Statistical_distance . $\endgroup$– Reinstate MonicaNov 18, 2014 at 21:42
-
$\begingroup$ Yes: see stats.stackexchange.com/questions/6907/… $\endgroup$– DidNov 18, 2014 at 22:20
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Beyond the symmetric KL-divergence, Information Theoretic Learning presented several symmetric distribution "distances". The idea is just to realize that pdfs are like any other functions in a L2-space. Thus, you can calculate the Euclidian distance $\int_x(p(x)-q(x))^2dx$, Cauchy-Schwarz distance, etc. There are even approximations to these distances directly from data, using Parzen Windows. Check the link above for more.
