I have two samples of probability distributions that I would like to compare. I have previously heard about the Kullback-Leibler divergence, but reading up on this it seems like its non-symmetricity makes it more suitable for comparing a sample to a model, rather than comparing two samples. What would you propose that I use instead, or maybe the KL divergence actually is a good choice?
Beyond the symmetric KL-divergence, Information Theoretic Learning presented several symmetric distribution "distances". The idea is just to realize that pdfs are like any other functions in a L2-space. Thus, you can calculate the Euclidian distance $\int_x(p(x)-q(x))^2dx$, Cauchy-Schwarz distance, etc. There are even approximations to these distances directly from data, using Parzen Windows. Check the link above for more.