# Does maximum likelihood estimation maximize probability really?

In maximum likelihood estimation (for continuous rv) we would like to maximize some parameter of our density function such that at some fixed observations our density function is bigger or equal for same density function bur for all other parameters. I would like to say that if we maximize the likelihood function, the probability that our observations occur is maximized as well. But, I can’t really produce a proof of this (I can’t convince myself that around each sample point I can maximize its probability by maximizing the density function). If someone could explain this I would appreciate it

• Yes, the likelihood function for some fixed observations $X_1=x_1,.,X_n=x_n (X_i$ randomv variables having joint density function) is given by the function $\theta\to f_{X_1,..,X_n}(x_1,..,x_n,\theta)$. So my question is how the maximum likelihood estimation maximizes the probability that $x_1,..,x_n$ occur? I know that if $\hat\theta$ is maximum, then $\forall \theta$ $f(x_1,..,x_n,\hat\theta)\ge f(x_1,..,x_n,\theta)$ and how could I conclude ? Dec 13, 2022 at 7:40
• The maximum likelihood estimator estimates the parameter from the observation by finding the parameter value which which would have maximised the conditional probability (or probability density) of the observation given the parameter, but it does not maximise the probability (or probability density) of the parameter given the observation. Whether the probability of the parameter given the observation is even a meaningful idea may depend on whether you take a Bayesian or frequentist approach. Dec 13, 2022 at 9:51

In some sense, no. If the density functions $$f_\theta$$ are continuous, for any fixed $$x\in\mathbb{R}^n$$ we have: $$\mathbb{P}_\theta\left[\bigcap_{i=1}^nX_i=x_i\right]\leq \mathbb{P}_\theta\left[X_1=x_1\right]=0$$ In other words, the probability of observing any specific values $$x_1,\ldots,x_n$$ is always zero. If you want, the MLE still maximizes the probability that the observed values occur (which is zero), but so does any other parameter estimate.

If your random variable only takes finitely many values, then: $$\mathbb{P}_\theta[X_1=x_1]=f_\theta(x_1,\theta)$$ So, in this case the MLE really gives the parameter $$\theta$$ which maximizes the probability of observing the data.

Edit: Here's another way to look at the MLE, if the distribution of the $$X_i$$ is continuous. We could choose the parameter $$\hat{\theta}_h$$ which maximizes the probability of observing values in a small ball $$(-h,h)$$ around our observations, for $$h>0$$: $$\hat{\theta}_h:=\arg\max_{\theta\in\Theta}\prod_{i=1}^n\mathbb{P}_{\theta}\left[ X_i\in(x_i-h,x_i+h) \right] =\arg\max_{\theta\in\Theta}\prod_{i=1}^n\frac{\mathbb{P}_{\theta}[ X_i\in(x_i-h,x_i+h)]}{2h}$$ Here I assumed for simplicity that the $$X_i$$ are independent. For any $$i$$, we can write this probability as: $$\frac{\mathbb{P}_{\theta}[ X_i\leq x_i+h]-\mathbb{P}_{\theta}[ X_i\leq x_i-h]}{2h}$$ $$=\frac{\mathbb{P}_{\theta}[ X_i\leq x_i+h]-\mathbb{P}_{\theta}[ X_i\leq x_i]}{2h} +\frac{\mathbb{P}_{\theta}[ X_i\leq x_i]-\mathbb{P}_{\theta}[ X_i\leq x_i-h]}{2h}$$ As $$h\rightarrow 0$$, $$\rightarrow \frac{f(x_i,\theta)}{2}+\frac{f(x_i,\theta)}{2}=f(x_i,\theta)$$ So, we can expect that $$\hat{\theta}_h\rightarrow\hat{\theta}_{MLE}$$ (this last step can be made rigorous, but you may need some regularity assumptions).

In words: In the continuous case, you could define an estimator which maximizes the probability that your observations lie in a small ball around the observations. The MLE is the limiting case of this estimator, as the size of these balls shrinks to zero.

• Alright, thank you. But so why is MLE used for continuous random variables? Like what are we maximizing exactly by maximizing the parameter? Dec 13, 2022 at 9:48
• Good question, I edited my answer to give one perspective. Dec 13, 2022 at 10:36
• Very nice answer, thank you very much! Dec 13, 2022 at 10:49

The definition of likelihood function is $$P(X| \theta)$$ , where $$X$$ is your observation and $$\theta$$ is your parameter, $$P$$ is the probability density function or probability mass function.

So in discrete case, likelihood is just probability of getting observation $$X$$ conditional on $$\theta$$. Likelihood in wiki

In continuous case, it is the PDF of $$X|\theta$$, which same as your "probability of getting particular observation" in a loose sense. (Since in a continuous case, probability of getting any partiuclar observation is always zero)

So maximize likelihood is by definition maximizing probability of getting particular observation.