Suppose that one has independent and identically distributed samples $x_i,i=1,...,n$ from some unknown density and one wants to fit a probability distribution $f_\theta(x)$, where $\theta$ is a (finite-dimensional) parameter, e.g. $\theta \in \mathbb{R}$, to that data. It seems that one way to estimate $\theta$ could be to minimize the Kullback-Leibler divergence between an estimate of pdf computed from data $x_i$ (using e.g. some kernel estimator) and the model $f_\theta$. I know that KL is used in many problems as a distance measure between two probability distributions (though it not actually a distance since it is not symmetric) but I don't think I have seen any theory about an estimator that it defined as a minimizer of this distance.

Also there is a connection to maximum likelihood estimation since if $p(x)$ is the pdf computed from the data we have \begin{align} KL(p || q) &= \int p(x) \log\frac{p(x)}{f_\theta(x)} dx = \int p(x) \log p(x) dx - \int p(x) \log f_\theta(x) dx \\ &= H(p) - E(\log f_\theta(x)),\end{align} where the first term (entropy of $p$) is constant and the latter term can be approximated with $\frac{1}{n}\sum_{i=1}^n \log f_\theta(x_i)$. So maximizing the last term (i.e. ML estimator) gives approximately the minimum KL divergence.

So my question is that is there any general theory about such estimator and some (not overly uncommon) applications where such method is used? Or is it so that such estimator typically has not-so-nice properties or is it just overrun by other estimation methods (like maximum likelihood, (generalized) method of moments etc.) that are likely easier to apply. Also I know there is some theory about minimum Hellinger distance estimation which seems to have nice efficiency and robustness properties but I would guess that is not the case with KL then?

  • $\begingroup$ This is actually the cost function used for logistic regression or more generally multinomial regression...or even more generally in classification neural networks, which are just affine transformations+sigmoid with a multinomial regressor tacked on the end. You'll often see the latter term in the KL called "cross entropy" in this setting. It can be derived from maximum likelihood as well, since taking the log doesn't change the maximizer. $\endgroup$ – Chester Aug 14 '15 at 19:59
  • 1
    $\begingroup$ @Chester Thank you for the note! Quite interesting but not exactly what I was looking for. That is, I want to know if there is similar theory (unbiasedness, asymptotic analysis etc.) that exists and is well-known for e.g. maximum likelihood and generalised method of moments estimators. $\endgroup$ – MarkoJ Aug 18 '15 at 15:22
  • $\begingroup$ I see. So, UMVUE, Fischer information, etc... in the context of KL divergence? $\endgroup$ – Chester Aug 18 '15 at 15:55
  • $\begingroup$ Yes, exactly, if such theory exists in the first place. $\endgroup$ – MarkoJ Aug 18 '15 at 15:57
  • $\begingroup$ Haven't you just shown that KL is equivalent to ML? So that all ML theory should carry over. $\endgroup$ – JohnRos Aug 21 '15 at 10:04

It is known that Min-KL and ML do coincide in the full esponential family. See, for example, here. In these cases, all ML theory applies.

In other cases, min-KL can still be seen as an M-estimator (a.k.a. Empirical Risk Minimizer). In which case, M-estimation theory applies.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.