Here is an excerpt from Pattern Recognition and Machine Learning by Christopher Bishop:

enter image description here

This seems to be not quite right—"the probability of the data set", when the data set is drawn from a Gaussian (or any continuous) distribution, is zero. Right?

The book goes on to explain how maximum likelihood estimation finds the parameters that maximize "the probability of the data". But that probability is always zero, so what are we actually maximizing?

It is clear that what the author means to say is that we are maximizing the joint probability density of the observations—that's actually what is expressed by $p(\mathbf{x}|\mu, \sigma^2)$ above. But that's a little unsatisfying to me—it would be nice to have a statement in terms of probability, not probability density.

I am thinking that maybe the correct way to put it is something like this: the maximum likelihood solution finds parameters such that there exists a neighborhood $M$ centered at $\mathbf{x}$ where for any neighborhood $N\subset M$ centered at $\mathbf{x}$, $P(\mathbf{x}\in N)$ is maximized.

But I just made that up. Is it true? Is it a good way to think about what maximum likelihood is actually finding?


You are correct, it is not probability of the data set. That is why it is called "the likelihood of the parameter given the data set". The maximum likelihood estimation is just one popular statistical method equipped with many optimal properties. But using likelihood is well enough to make statistical inference for almost any problem you investigate (see,Casella "Statistical inference" 2nd, section 6.3). Another deeper understanding lies in measure theory, for continuous random variable, there is no way to assign valid probability mass that satisfies the axiom of probability to each data point. As an alternative using Caratheodory's theroem one can construct probability measure on sigma-algebra (a class of sets). If you really want to use probability argument directly, you may want to take a look at Bayesian inference. Just some of my thoughts


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.