To avoid answers too complicated for me to understand/to clear notation, I'm describing quickly the setting in which everything takes place:

In our intro statistic lecture the following we said that the following components made up an estimation problem

  • an at most countable space $\mathcal{X}$ of all possible samples we can observe

  • a family $(P_\theta)_{\theta \in \Theta} $ of distributions on $\mathcal{X}$

  • a function $f:\Theta \rightarrow \mathbb{R}$ that we would like to estimate

We defined the maximum-likelihood estimator $\hat{\theta}_x$ as maximal $\theta$ that is attained by the function $P_\theta(x)$, if such a maximum exists and the expectancy of any estimator $S$ as $E_\theta S:=\sum_{x\in \mathcal{X}} S(x)_\theta (x)$. Is it then true that

  • $\boldsymbol{\hat{\theta}_x=\sup\{ P_\theta (x) \mid \theta \in \Theta \} }$ ?

  • $\boldsymbol{E_\theta \,\hat{\theta}_x =\hat{\theta}_x}$ ? I think this holds, since $\hat{\theta}_x$ is just a number in the unit interval and the expectancy of a number is that number.
    This would mean that the ML is always biased, since it doesn't hold that for all $\theta \in \Theta$ we have $E_\theta \,\hat{\theta}_x =f(\theta)$ -- except when $f$ maps everything to $\hat{\theta}_x$ which seems odd.

  • 4
    $\begingroup$ Counterexample: the sample mean of a gaussian is the ML estimator of the mean. And it's unbiased. $\endgroup$ – leonbloy Jan 2 '13 at 14:29
  • 1
    $\begingroup$ @leonboy that is too complicated for me, since I learned about all of this just a couple of days ago. you need to take me be the hand and tell me what I did wrong :(, since I can't understand other stuff than what we did in the first lectures (which is roughly what I described above) $\endgroup$ – juhrin Jan 2 '13 at 14:34

The simplest situation does the job... Assume that one throws $n$ times a biased coin which has probability $\theta$ of landing on heads and probability $1-\theta$ of landing on tails, for some $\theta$ in $(0,1)$. One records the results of these throws as $x=(x_1,x_2,\ldots,x_n)$, where $x_k=1$ if the $k$th throw results in heads and $x_k=0$ if it results in tails. The likelihood $P_\theta(x)$ of the whole experience is $$ P_\theta(x)=\theta^{s}\cdot(1-\theta)^{n-s},\qquad s=\sum_{k=1}^nx_k, $$ which is maximum at $$ \hat \theta=\frac{s}n. $$ (Note that $\frac{\mathrm dP_\theta(x)}{\mathrm d\theta}=\left(\frac{s}\theta-\frac{n-s}{1-\theta}\right)\cdot P_\theta(x)=\frac{s-n\theta}{\theta(1-\theta)}\cdot P_\theta(x)$ hence the function $\theta\mapsto P_\theta(x)$ is increasing on $\theta\leqslant\frac{s}n$ and decreasing on $\theta\geqslant\frac{s}n$.)

One should be able to show that in this situation $\mathbb E_\theta(\hat \theta)=\theta$ for every $\theta$, making the MLE $\hat \theta$ unbiased, uniformly on $\theta$.


Your problem is that it doesn't make sense to say $\mathbb E \hat\theta_x = \hat\theta_x$.
Your estimator $\hat\theta_x$ is a function of the data, while its expected value must be a function of the parameters. So you have $\mathbb E \hat\theta_x = g(\theta)$ for some function $g$. Your estimator is unbiased iff $g(\theta) \equiv f(\theta)$.

The counter-example to your claim provided by leonbloy is as follows: If we're trying to estimate the mean of a sample of from a Normal (gaussian) distribution, the maximum-likelihood estimator is $\frac1n\sum x_i$, which is of course unbiased. So clearly maximum-likelihood estimators are not always biased.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.