Is the maximum-likelihood estimator always biased?

To avoid answers too complicated for me to understand/to clear notation, I'm describing quickly the setting in which everything takes place:

In our intro statistic lecture the following we said that the following components made up an estimation problem

• an at most countable space $\mathcal{X}$ of all possible samples we can observe

• a family $(P_\theta)_{\theta \in \Theta}$ of distributions on $\mathcal{X}$

• a function $f:\Theta \rightarrow \mathbb{R}$ that we would like to estimate

We defined the maximum-likelihood estimator $\hat{\theta}_x$ as maximal $\theta$ that is attained by the function $P_\theta(x)$, if such a maximum exists and the expectancy of any estimator $S$ as $E_\theta S:=\sum_{x\in \mathcal{X}} S(x)_\theta (x)$. Is it then true that

• $\boldsymbol{\hat{\theta}_x=\sup\{ P_\theta (x) \mid \theta \in \Theta \} }$ ?

• $\boldsymbol{E_\theta \,\hat{\theta}_x =\hat{\theta}_x}$ ? I think this holds, since $\hat{\theta}_x$ is just a number in the unit interval and the expectancy of a number is that number.
This would mean that the ML is always biased, since it doesn't hold that for all $\theta \in \Theta$ we have $E_\theta \,\hat{\theta}_x =f(\theta)$ -- except when $f$ maps everything to $\hat{\theta}_x$ which seems odd.

-
Counterexample: the sample mean of a gaussian is the ML estimator of the mean. And it's unbiased. –  leonbloy Jan 2 '13 at 14:29
@leonboy that is too complicated for me, since I learned about all of this just a couple of days ago. you need to take me be the hand and tell me what I did wrong :(, since I can't understand other stuff than what we did in the first lectures (which is roughly what I described above) –  juhrin Jan 2 '13 at 14:34

Your problem is that it doesn't make sense to say $\mathbb E \hat\theta_x = \hat\theta_x$.
Your estimator $\hat\theta_x$ is a function of the data, while its expected value must be a function of the parameters. So you have $\mathbb E \hat\theta_x = g(\theta)$ for some function $g$. Your estimator is unbiased iff $g(\theta) \equiv f(\theta)$.
The counter-example to your claim provided by leonbloy is as follows: If we're trying to estimate the mean of a sample of from a Normal (gaussian) distribution, the maximum-likelihood estimator is $\frac1n\sum x_i$, which is of course unbiased. So clearly maximum-likelihood estimators are not always biased.
The simplest situation does the job... Assume that one throws $n$ times a biased coin which has probability $\theta$ of landing on heads and probability $1-\theta$ of landing on tails, for some $\theta$ in $(0,1)$. One records the results of these throws as $x=(x_1,x_2,\ldots,x_n)$, where $x_k=1$ if the $k$th throw results in heads and $x_k=0$ if it results in tails. The likelihood $P_\theta(x)$ of the whole experience is $$P_\theta(x)=\theta^{s}\cdot(1-\theta)^{n-s},\qquad s=\sum_{k=1}^nx_k,$$ which is maximum at $$\hat \theta=\frac{s}n.$$ (Note that $\frac{\mathrm dP_\theta(x)}{\mathrm d\theta}=\left(\frac{s}\theta-\frac{n-s}{1-\theta}\right)\cdot P_\theta(x)=\frac{s-n\theta}{\theta(1-\theta)}\cdot P_\theta(x)$ hence the function $\theta\mapsto P_\theta(x)$ is increasing on $\theta\leqslant\frac{s}n$ and decreasing on $\theta\geqslant\frac{s}n$.)
One should be able to show that in this situation $\mathbb E_\theta(\hat \theta)=\theta$ for every $\theta$, making the MLE $\hat \theta$ unbiased, uniformly on $\theta$.