# Maximum Likelihood Estimation and bias

I have a strange (well to me at least) MLE problem. If we let $\{X_i\}_{i=1}^n$ be an i.i.d. sample of a random variable $X$ whose mean is $\mu$ and variance $\sigma^2$. Suppose further that $X_1\sim N(\mu,1)$. I must show that the MLE, $\max\{\bar{X}_n,1\}$for $\max\{\mu,1\}$ suffers from bias.

First of all, what does $\max\{\bar{X}_n,1\}$for $\max\{\mu,1\}$ mean?

-
First you say it's an i.i.d. sample of a random variable whose mean is $\mu$ and whose variance is $\sigma^2$. Then you say $X_1\sim N(\mu,1)$. Why mention $\sigma^2$ and then obliquely say that $\sigma^2=1$? Why not just say it's an i.i.d. sample from $N(\mu,1)$? –  Michael Hardy Oct 2 '12 at 23:59
You seem to assert that $\max\{\bar{X}_n,1\}$ is the MLE for $\max\{\mu,1\}$. Why is that? –  Did Oct 3 '12 at 9:33

(a) when $\mu \lt 1$ and $\max\{\mu,1\}=1$, $\max\{\bar{X},1\}$ is sometimes strictly greater than $1$ (since $\bar{X} \gt 1$ with positive probability) and is never less than $1$, so $$E[\max\{\bar{X},1\}] \gt 1 = \max\{\mu,1\}$$ meaning there is upward bias.

(b) when $\mu \gt 1$ and $\max\{\mu,1\}=\mu$, $\max\{\bar{X},1\}$ is sometimes strictly greater than $\bar{X}$ (since $\bar{X} \lt 1$ with positive probability) and is never less than $\bar{X}$, so $$E[\max\{\bar{X},1\}] \gt E[\bar{X}] = \mu =\max\{\mu,1\}$$ meaning there is upward bias.

For there to be no bias, you need either $\Pr(\bar{X} \le 1) =1$ or $\Pr(\bar{X} \ge 1) =1$, but this is not the case with a normal distribution.

-

First I answer your first question while thinking the rest. You have $\max\{\overline{X},1\}$ as an estimator. That is, you find the sample mean of your data which is supposed to be $\overline{X}$ and take the maximum of $\overline{X}$ and $1$.

This is called your estimator. I havent checked if it is a maximum likelihood estimator yet. If so, it should minimize the likelihood function.

Now you have an estimator (probabily maximum likelihood) which is trying to estimate the maximum of the $\mu$, the parameter of your distribution and $1$.

EDIT: Okay here is the idea: You want to estimate the maximum of $\mu$ and $1$. There might be two cases. Either $\mu>1$ or $\mu<1$. In the first case $\max\{\mu,1\}=1$ else is $\mu$.

Assume that $\mu<1$ then we have $\max\{\mu,1\}=1$. This means our objective is to estimate $1$ from given samples. Now lets have a look at our estimator. This estimator will give you less than half of the times the value that you have and else you will get $1$. When you check the expected value then it will be something less than $1$ which indicates a Bias towards the positive real axis.

On the other hand when $\mu>1$, we have $\max\{\mu,1\}=\mu$. But we have real problem! our estimator will output more than half of the time a number which is greater than $1$ and less than half of the time a value which is less than $1$ which will be eventually rounded to $1$ due to its structure $\max\{\mu,1\}$. As you can see, we want to estimate $\mu$ and unfortunately all the values which are negative and between $\mu$ and $1$ are rounded to $1$. That is not good! this will result a Bias towards the positive real axis. Because in order not to have a Bias, we need to have equally from the values that are less than $\mu$ and greater than $\mu$ in the order. Not the time of occurance. As all the negative values will be mapped to $1$, this estimator will never converge to $\mu$ but something greater than $\mu$.

To show this theoretically. You need to determine the expected value of this estimator in both cases and show that it deviates from the true (to be estimated) value.

-
I would have thought the estimator was biased upwards in both cases –  Henry Oct 3 '12 at 7:34
@Henry true. misdeclination is corrected. –  Seyhmus Güngören Oct 3 '12 at 8:51

Taking expectations over the inequality $\max\{\bar X_n,1\}-1\geq 1_{(\bar X_n\geq 2)}$ gives $$\mathbb{E}(\max\{\bar X_n,1\})-1\geq \mathbb{P}(\bar X_n\geq 2)>0,$$ or $$\mathbb{E}(\max\{\bar X_n,1\})>1.\tag1$$

Similarly, taking expectations over the inequality $\max\{\bar X_n,1\}-\bar X_n\geq 1_{(\bar X_n\leq 0)},$ gives $$\mathbb{E}(\max\{\bar X_n,1\})-\mathbb{E}(\bar X_n)\geq \mathbb{P}(\bar X_n\leq 0)>0,$$ or $$\mathbb{E}(\max\{\bar X_n,1\})>\mathbb{E}(\bar X_n).\tag2$$

Combining (1) and (2) gives $$\mathbb{E}(\max\{\bar X_n,1\})>\max\{1,\mathbb{E}(\bar X_n)\}$$ so $\max\{\bar X_n,1\}$ is a biased estimator of $\max\{1,\mathbb{E}(\bar X_n)\}$.

The probabilities $\mathbb{P}(\bar X_n\geq 2)$ and $\mathbb{P}(\bar X_n\leq 0)$ are strictly positive because $\bar X_n$ has a normal distribution.

-

To ask what "$\max\{\bar{X}_n,1\}$for $\max\{\mu,1\}$" means is to parse the sentence incorrectly. It's asking about $\max\{\bar X_n, 1\}$, which is the MLE for $\max\{\mu,1\}$.

First, do you know how to show that $\bar X_n$ is the MLE for $\mu$? MLEs generally are equivariant under more-or-less everything, i.e. if $\bar X_n$ is the MLE for $\mu$, then for every function $g$, then $g(\bar X_n)$ is the MLE for $g(\mu)$.

So let $g(x)=\max\{x,1\}$. Then conclude that $\max\{\bar X_n, 1\}$ is indeed the MLE for $\max\{\mu,1\}$.

If $\mu>1$ then $\mathbb{E}\bar X = \mu$, and $\Pr(\max\{\bar X_n,1\}> \bar X_n)>0$ and $\Pr(\max\{\bar X_n,1\} < \bar X_n)=0$, and consequently $\mathbb{E}\max\{\bar X_n,1\}>\mathbb{E}\bar X_n =\mu$. Thus the estimator is biased.

-
thanks for the help @Michael Hardy why knowing $g(x) = max\{x,1\}$ is relevant for this problem? Is it to simply emphasize the equivalency of $\bar{X}_n=\mu$? And is it possible to have that $Pr(max\{\bar{X}_n,1\}=\bar{X}_n)$ even if $\mu>1$? –  ChuckM Oct 3 '12 at 3:48
The reason it's relevant is that that's what tells you that if $\bar X$ is the MLE of $\mu$, the $\max\{\bar X,1\}$ is the MLE of $\max\{\mu,1\}$. –  Michael Hardy Oct 3 '12 at 3:52
This begs for a clear proof of the assertion that $\max\{\bar{X}_n,1\}$ is indeed the MLE for $\max\{\mu,1\}$. –  Did Oct 3 '12 at 9:35
The way to prove that is to prove a more general assertion that if $T$ is the MLE for $\theta$ then $g(T)$ is the MLE for $g(\theta)$. This is in every book that introduces the theory of statistics. –  Michael Hardy Oct 3 '12 at 15:38
Sure it is. As sure as the fact that this will not help the OP since, the function $\mu\mapsto\max\{\mu,1\}$ being many-to-one, the usual theorem needs some reformulation. (Anyway, please try to use the @ notification system, unless you want people to miss that you replied to their comment.) –  Did Oct 6 '12 at 15:09