# metrics for density-sampling similarity, beyond likelihood

I am looking for a metric that would evaluate the distance between a sample $S$ and a density function $D$

Building a sample from a known distribution can be done using a monte-carlo sampling, however the invers operation is much harder. I am studying different methods, and impact of errors happening in those algorithms, therefore I need a distance between those two objects.

Maximum likelihood is not something that meets my needs. To show why let's take the exemple of a gaussian distribution centered on $0$. Maximum likelihood whould be achieved by having all my point at point $0$, any point in a different place would correspond to a lower probability and therefore reduce the total likelihood.

For me, minimum distance should be achieved if for all interval $I'$ of my domain $I$ I have something like $$\int_{I'} D(x) \mathrm dx = \frac{\|\{x\in S | x\in I'\}\|}{\|S\|}$$ That means that however low the density is in an interval, if the interval is large enough (or if my sample contains enough point) one or more point are expected to be in it.

Is there a metric that would have the property i'm looking for ?

How about taking the Kullback-Leibler divergence between the empirical distribution (of the sample), and the target distribution?

• This Kullback-Leibler looks very interesting ! It looks like it is a metric between two distributions and, while it can be usefull to me, I'm looking more for a metric between a distribution and a sampling (the sampling being a set of point). Should a consider that my sampling is a sum of diracs with height $1/n$ with n the number of points ?
– Amxx
Jan 24, 2015 at 1:29
• @amxx yep that's what I was thinking you could do. Turn your sample into its equivalent distribution
– user76844
Jan 24, 2015 at 3:46

Here are some suggestions:

First calculate the histogram of your data. Then discretize your density function. After that use a discrete version of some distance. It can even be the Euclidean distance, KL- divergence, or Hellinger distance. There are many of them.

Another way is to consider kernel density estimation (KDE). After than you can use the continuous version of the any distance of your interest. Here what matters is the quality of your KDE.

• My data consist in set of 3D points and my density function is the results of a KDE (with adaptive windows size), which is already discretized over a grid. I'll have a look at KL-divergence and Hellinger distance. Basically I don't know the truth, I just know that some errors might during my KDE (because of memory corruption I try to modelize), that's why I need to know how well my resulting density fits my data points
– Amxx
Jan 24, 2015 at 1:48