0
$\begingroup$

Suppose I observe the outcomes ($x_1$ through $x_n$) of $n$ rolls ($X_1$ through $X_n$) of a fair, $\theta$-sided die, and want to find a point estimate using Maximum Likelihood Estimation. I will state the problem without referring to the likelihood function in order to eliminate one layer of confusion.

In the mindset of modeling the problem, it seems to me that the goal is to choose the possible value of $\theta$ which has the greatest probability of being its actual value, given the signals generated by the roll information, and that this goal in mathematical notation is $$\max_\theta \Pr(\theta\ |\ (\bigcap_{i = 1}^n X_i = x_i))$$ However, in the course of solving the problem, the the formula used is $$\max_\theta \Pr((\bigcap_{i = 1}^n X_i = x_i)\ |\ \theta)$$

Is the point of the maximum likelihood principle that both of these maximizations produce the same answer? If not, then why is the latter the correct one to solve, even while the former seems to better capture the English description of the problem statement?

$\endgroup$
7
  • $\begingroup$ Hi: I think you understand the MLE concept based on what you wrote. But you can't use the second expression to find the MLE because, in that expression, $\theta$ is given and $\theta$ is the unknown when doing maximum likelihood. Note that the second expression could be used if you want to some kind of grid search for different values of $\theta$. Still, a more direct approach is to use the first expression which will result in an expression where $\theta$ is unknown and the likelihood is then maximized with respect to $\theta$. $\endgroup$
    – mark leeds
    Apr 20 at 4:06
  • $\begingroup$ @markleeds "But you can't use the second expression to find the MLE because, in that expression, $\theta$ is given and $\theta$ is the unknown when doing maximum likelihood." This is entirely incorrect. See equation (2) of heropup's answer. $\endgroup$
    – snar
    Apr 20 at 4:46
  • $\begingroup$ @user10478: The last term in Equation (1) in snar's answer is EXACTLY what you want to maximize ( except you would take the log to make the maximization easier ). You and I were both sloppy in terms of our use of probability, likelihood and density so I won't even attempt it to give it a name. Otherwise, I might be "ENTIRELY" incorrect. LOL. $\endgroup$
    – mark leeds
    Apr 20 at 15:10
  • $\begingroup$ Also, note that heropup's answer is quite nice and provides beautiful explanation in terms of the usage of density, likelihood etc. I just wanted to point out that his (1) is what you would maximize. heropup: Thanks for your detailed explanation. $\endgroup$
    – mark leeds
    Apr 20 at 15:14
  • 1
    $\begingroup$ @user10478 As others have pointed out, from a frequentist perspective, it does not make sense to think about it as “which value is most probable.” Your question, however, is very natural, and is the fundamental question statistical decision theory aims to answer. You should look into it if you want a reasonable resolution. $\endgroup$
    – Andrew
    Apr 22 at 2:23

1 Answer 1

Reset to default
2
$\begingroup$

The first equation is problematic in the frequentist viewpoint. To illustrate, let us consider the familiar model $$X_i \mid \theta \sim \operatorname{Bernoulli}(\theta), \\ \Pr[X_i = 1] = \theta, \quad \Pr[X_i = 0] = 1 - \theta.$$ Given a sample $(x_1, \ldots, x_n)$, the joint probability is $$\prod_{i=1}^n \Pr[X_i = x_i \mid \theta] = \prod_{i=1}^n \theta^{x_i} (1-\theta)^{1-x_i} = \theta^{\sum x_i} (1 - \theta)^{n - \sum x_i}, \tag{1}$$ where $\sum x_i$ is the sample total, or equivalently, the number of observations that equal $1$.

But the problem with $(1)$ is that this is not a probability density over the parameter space $\theta \in [0,1]$. To be specific, it is proportional to a density, but $$\int_{\theta=0}^1 \theta^{\sum x_i} (1 - \theta)^{n - \sum x_i} \, d\theta \ne 1$$ for general $n$ and $\sum x_i$. This is why a statement like $$\Pr\left[\theta \;\left| \;\bigcap_{i=1}^n X_i = x_i \right.\right]$$ is, strictly speaking, not quite correct unless we are talking about a Bayesian posterior for $\theta$, in which case a prior distribution for $\theta$ will need to be specified. Moreover, even if we allow a Bayesian interpretation, $\Pr[\theta \mid \cdot]$ is also problematic in the case where $\theta$ is a parameter with continuous support; instead, we would need to write $f(\theta \mid \cdot)$.

In the context of maximum likelihood estimation, it is simply better to dispense with all of this and describe the quantity to be maximized as $\mathcal L(\theta \mid \cdot)$, a likelihood function of $\theta$ with respect to some sample or observed outcome. The fact that the likelihood is proportional to the joint density, i.e. $$\mathcal L(\theta \mid x_1, \ldots, x_n) \propto f_{X_1, \ldots, X_n}(x_1, \ldots, x_n \mid \theta), \tag{2}$$ is, for all intents and purposes, a definition. Likelihoods are not unique; they are unique up to a (positive) constant of proportionality. It is because of $(2)$ that we are able to use the joint density to maximize the likelihood, since knowing the density as a function of the parameter $\theta$ allows us to choose the "most likely" $\theta$ that generated the sample. Note also that there are nontrivial cases for which "most likely" is not unique; i.e., there can be more than one MLE for a sample.

$\endgroup$
5
  • $\begingroup$ This is insightful, but I am still a bit confused. While modeling a real world problem such as the die example, candidate answers would be "it is more probable that the die has 5 sides than any other number," and "it is more probable that the die has 9 sides than any other number." These are statements about probabilities of the population, given a sample, in line with the first expression in my post. On the other hand, the solution technique manipulates the second expression in my post, probabilities of a sample, given the population. Framing in terms of likelihoods seems to obfuscate $\endgroup$
    – user10478
    Apr 20 at 16:50
  • $\begingroup$ the confusion, because likelihoods and probabilities have the same colloquial meaning, but differ in the technical sense in which you've defined them. $\endgroup$
    – user10478
    Apr 20 at 16:52
  • $\begingroup$ @user10478 When you make a statement like "it is more probable that the die has 5 sides..." that is a colloquial use of the word "probable" and not a mathematical one, because the mathematical notion of probability requires a well-defined probability space, and in your case, no such space is defined for the number of sides of the die. "Probabilities of the population" makes no sense. The only thing that has a probability here is the outcome of the die conditioned on the total number of sides it has. The number of sides is a parameter which is fixed but unknown. $\endgroup$
    – heropup
    Apr 21 at 23:22
  • $\begingroup$ "Likelihood" and "probability" may have the same colloquial meaning, but they are distinct (but related) mathematical concepts. This is why we call it "maximum likelihood estimation" and not "maximum probability estimation." It is necessary to characterize the inference in terms of maximization of a likelihood function because no probability measure has been defined on the parameter for which estimation is desired, and in the frequentist view, it is nonsensical to do so because the parameter is fixed. $\endgroup$
    – heropup
    Apr 21 at 23:29
  • $\begingroup$ I would have called the probability that the die has 5 sides an epistemic probability. I think this is roughly what you mean by the Bayesian interpretation. A game of chess has no randomness, but there are still epistemic probabilities that opponents will freely choose particular strategies. $\endgroup$
    – user10478
    Apr 21 at 23:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .