Properties of minimizing statistical distance I have some data, thus I have its empirical distribution. I want to use a theoretical distribution to fit my data. For example, I observe my data is likely to be distributed as Pareto, so I use Pareto to fit. But the major point is to estimate the parameters in Pareto, so I tried MLEs. Also I tried to minimize KLD which can be proved to be equivalent to MLEs. I am wondering if I want to minimize some other probability distance, are there any beautiful properties for that? I found some measures on the distances here https://en.wikipedia.org/wiki/Statistical_distance. 
For instance, minimizing one of the distance, I will get an estimator whose expectation is just the true parameter asymptotically. 
 A: Your problem falls under the classical setting of inverse problems; problems where you are required to learn a probability distribution, given data samples picked from the underlying alphabet. Typically, certain features of the data samples are measured, and then an attempt is made to fit a distribution that conforms to the features measured. 
For example, you could measure the mean and variance of the data points you have, and then ask: what is the distribution that I can fit to my data points such that the mean and variance computed under this distribution are what were measured, but otherwise, the distribution is as random as possible? Here, the randomness of a distribution is measured by means of Shannon entropy of the distribution.
The principle explained in the above paragraph is termed as the maximum (Shannon) entropy principle. A lot of work has gone into this. Perhaps, the best resource you could look up in order to understand more about this is Prof. Sanjoy Dasgupta's talk on Information Geometry, which you may find here:
http://videolectures.net/mlss05us_dasgupta_ig/
It turns out that exponential family of distributions do the right fitting; the parameters of the exponential family depending on the measurements being made. The right procedure to be followed in order to estimate the parameters of the exponential family are briefly outlined in the above lecture. Hope you find this useful.
