Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

To avoid answers too complicated for me to understand/to clear notation, I'm describing quickly the setting in which everything takes place:

In our intro statistic lecture the following we said that the following components made up an estimation problem

  • an at most countable space $\mathcal{X}$ of all possible samples we can observe

  • a family $(P_\theta)_{\theta \in \Theta} $ of distributions on $\mathcal{X}$

  • a function $f:\Theta \rightarrow \mathbb{R}$ that we would like to estimate


We defined the maximum-likelihood estimator $\hat{\theta}_x$ as maximal $\theta$ that is attained by the function $P_\theta(x)$, if such a maximum exists and the expectancy of any estimator $S$ as $E_\theta S:=\sum_{x\in \mathcal{X}} S(x)_\theta (x)$. Is it then true that

  • $\boldsymbol{\hat{\theta}_x=\sup\{ P_\theta (x) \mid \theta \in \Theta \} }$ ?

  • $\boldsymbol{E_\theta \,\hat{\theta}_x =\hat{\theta}_x}$ ? I think this holds, since $\hat{\theta}_x$ is just a number in the unit interval and the expectancy of a number is that number.
    This would mean that the ML is always biased, since it doesn't hold that for all $\theta \in \Theta$ we have $E_\theta \,\hat{\theta}_x =f(\theta)$ -- except when $f$ maps everything to $\hat{\theta}_x$ which seems odd.

share|improve this question
4  
Counterexample: the sample mean of a gaussian is the ML estimator of the mean. And it's unbiased. –  leonbloy Jan 2 '13 at 14:29
1  
@leonboy that is too complicated for me, since I learned about all of this just a couple of days ago. you need to take me be the hand and tell me what I did wrong :(, since I can't understand other stuff than what we did in the first lectures (which is roughly what I described above) –  juhrin Jan 2 '13 at 14:34

2 Answers 2

Your problem is that it doesn't make sense to say $\mathbb E \hat\theta_x = \hat\theta_x$.
Your estimator $\hat\theta_x$ is a function of the data, while its expected value must be a function of the parameters. So you have $\mathbb E \hat\theta_x = g(\theta)$ for some function $g$. Your estimator is unbiased iff $g(\theta) \equiv f(\theta)$.

The counter-example to your claim provided by leonbloy is as follows: If we're trying to estimate the mean of a sample of from a Normal (gaussian) distribution, the maximum-likelihood estimator is $\frac1n\sum x_i$, which is of course unbiased. So clearly maximum-likelihood estimators are not always biased.

share|improve this answer

The simplest situation does the job... Assume that one throws $n$ times a biased coin which has probability $\theta$ of landing on heads and probability $1-\theta$ of landing on tails, for some $\theta$ in $(0,1)$. One records the results of these throws as $x=(x_1,x_2,\ldots,x_n)$, where $x_k=1$ if the $k$th throw results in heads and $x_k=0$ if it results in tails. The likelihood $P_\theta(x)$ of the whole experience is $$ P_\theta(x)=\theta^{s}\cdot(1-\theta)^{n-s},\qquad s=\sum_{k=1}^nx_k, $$ which is maximum at $$ \hat \theta=\frac{s}n. $$ (Note that $\frac{\mathrm dP_\theta(x)}{\mathrm d\theta}=\left(\frac{s}\theta-\frac{n-s}{1-\theta}\right)\cdot P_\theta(x)=\frac{s-n\theta}{\theta(1-\theta)}\cdot P_\theta(x)$ hence the function $\theta\mapsto P_\theta(x)$ is increasing on $\theta\leqslant\frac{s}n$ and decreasing on $\theta\geqslant\frac{s}n$.)

One should be able to show that in this situation $\mathbb E_\theta(\hat \theta)=\theta$ for every $\theta$, making the MLE $\hat \theta$ unbiased, uniformly on $\theta$.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.