# Consistency for maximum likelihood estimator with a single sample

Suppose you have a finite family of probability measures $$\{\mu_\theta: \theta \in S\}$$ on a finite space $$\Omega$$ (with respect to the discrete sigma algebra). Let $$X$$ be a random element of $$\Omega$$ distributed according to $$\mu_{\theta_0}$$, for some fixed $$\theta_0 \in \Omega$$. Consider the maximum likelihood estimator for $$\theta_0$$:

$$\hat{\theta}_{MLE} = \arg\max_\theta L(\theta | X)$$.

Here $$L$$ is the usual likelihood,

$$L(\theta | X) = \mu_\theta(X)$$.

I'm curious about consistency for the MLE in this simple setup, but not in the usual sense of 'asymptotic consistency:' rather, I want to know if there are some conditions under which the MLE is consistent for a single sample, i.e.

$$\arg \max_\theta \mathbb{P}(\hat{\theta}_{MLE} = \theta) = \theta_0$$.

(Sorry if my notation is a bit weird -- let me know if it's confusing and I'll try to clarify.) I am guessing this can fail in general -- is there a simple example? Are there some conditions under which it holds?

It is true that the expected value of $$L(\theta|X)$$ is maximized at $$\theta = \theta_0$$. Indeed, by Cauchy-Schwarz:

$$\mathbb{E}L(\theta | X) = \mathbb{E}\mu_\theta(X) = \sum_\omega \mu_\theta(\omega) \mu_{\theta_0}(\omega) \leq \sqrt{\sum_\omega \mu_\theta(\omega)^2 \sum_\omega \mu_{\theta_0}(\omega)^2}$$

Equality occurs when $$\theta = \theta_0$$. But this doesn't imply that the density of the MLE is maximized there.

Edit 1: To be careful, the conclusion should be that $$\theta_0$$ is a point that maximizes the density of the MLE (not necessarily the only point).

• I believe this can true only when the distributions of $X$ corresponging to different values of $\theta$ are orthogonal. In fact, the usual consistency can be regarded in this manner. For example, a sequence of i.i.d. $N(\mu,1)$ observations may be regarded as a single element of $\mathbb{R}^\infty$. Then the measures on $\mathbb R^\infty$ corresponding to different values of $\mu$ are orthogonal (e.g. since by SLLN $\frac1n(X_1+\dots + X_n)\to \mu$ a.s.). So it is possible to determine the value of $\mu$ from "single" observation (the likelihood of this value being 1, and of others, 0). – zhoraster Feb 20 at 10:04

Consider $$\theta$$ taking 3 possible values (0,1,2) and $$X$$ taking 2 possible values (0,1). We take: $$\mu_0 = (1/2,1/2)$$, $$\mu_1 = (0,1)$$, $$\mu_2 = (1,0)$$.
$$\hat\theta_{MLE}(X) = \cases{1 \quad\text{if}\quad X=0 \\ 2 \quad\text{if} \quad X=1\\}$$
$$\mathbb{P}(\hat\theta_{MLE} = 0) = 0$$