# When does a maximum likelihood estimate fail to exist?

I have been told that a maximum likelihood estimate (MLE) does not always actually exist. Why is this the case? It is clear that the MLE may not be unique, but there should always be a maximum, no?

-
–  Tim Nov 26 '11 at 17:36

I can think of the following example were MLE does not exist, in the sense that the estimation diverges. Let's say you have a few samples $x_1,...,x_n$ and you want to model that with a mixture of Gaussians $p(x) = \sum_{k=1}^K \pi_k \mathcal N(x|\mu_k, \sigma_k)$, where the $\pi_k$ sum to one.

In principle, you could just derive the log-likelihood of all the examples and optimize that with respect to parameters via gradient descent (which would be straightforward MLE). However, during the optimization, it might happen that one of the Gaussians sits exactly on one of the training examples. What the optimization would do in that case is to make the $\sigma$ of that Gaussian smaller and smaller, which increases the likelihood every time (basically because we have an "infinitely" narrow and high Gaussian sitting on one example). If you only had one Gaussian, the other examples would become less likely which would enforce a trade-off at some point. However, since the other Gaussians of the mixture give the remaining examples some finite likelihood, it does not matter that we make one single Gaussian narrower. Therefore, the likelihodd will diverge.

-

The MLE exists if the parameter space is compact and the Likelihood function is continuous on the parameter space.

It is unique if the parameter space is convex and the likelihood function is concave.

Edit

I should add that the above conditions are sufficient conditions.

-
But not only if the parameter space is compact. And that's pretty important, since probably most of the cases in which people find MLEs are cases in which the parameter space is not compact. –  Michael Hardy Nov 26 '11 at 15:48
I'm not familiar with the term 'compact'. Can you clarify? My MLE experience is generally in terms of location estimating for tracking applications :) –  blnakets Nov 26 '11 at 16:24

• Errors have expectation $0$;