# Maximum likelihood estimation and efficiency

Let $X_1, ..., X_n$ be independent observations of a random variable $X$ that has with probability $p$ the distribution $\mathcal{U}[0,a]$ and with probability $1-p$ the distrbiution $\mathcal{U}[0,b]$ where $a$ and $b$ are known with $a<b$.

1. What is the maximum-likelihood estimator for $p$?

2. Is that estimator efficient? (This means, is the Cramér–Rao bound achieved?)

I'm having quite some trouble with this task for several hours now. I think the density of $X$ is given by $$f_X(x) = p \frac{1}{a}1_{[0,a]}(x) + (1-p) \frac{1}{b}1_{[0,b]}(x).$$

For the trivial case $n=1$ this gives $\hat{p}=1_{[0,a]}(x_1)$ where the expected value of $X_1$ is $a/b$. So the estimator is in fact biased. So this doesn't really help with the second task.

I can't even figure out the general likelihood function. I think it should be something like $L(p) = (p \frac{1}{a})^k ... + ((1-p) \frac{1}{b})^m ...$ where $k$ denotes the number of values from the sample $x_1, ..., x_n$ where $x_i\le a$ and $m=n-k$. But I don't understand how to deal with the indicator functions and if maybe I have to add a factor like with the binomial distribution.

However, I took the raw form $L(p) = (p \frac{1}{a})^k + ((1-p) \frac{1}{b})^{(n-k)}$, $k$ like explained above and tried to maximize by using the log-likelihood. But from that I got $\hat p = \frac{an-bN}{an-bn}$ which doesn't strike me as correct as because of $a<b$ it's negative for small $N$. Nonetheless, I tried to calculate the Fisher information and compared it's inverse to the variance of $\hat p$ - I wasn't very surprised not to get an equality...

Could be that I messed up some calculations, too - dealing with the derivatives of the log-likelihood was a little messy. Now I'm rather frustrated.

Can anyone show me how to solve this?

-

If the indicators functions confuse you, don't use them. Write instead

$$f_X(x)=\begin{cases} \begin{array}{lc} {p \over a} + \frac{1-p}{b} & x\in (0,a)\\ \frac{1-p}{b} & x\in (a,b) \end{array}\\ \end{cases}$$

which, letting $A=1/a$ $B=1/b$ can also be written as

$$f_X(x)=\begin{cases} \begin{array}{lc} p\, (A-B) +B & x\in (0,a)\\ (1-p) \, B & x\in (a,b) \end{array}\\ \end{cases}$$

If you have $n$ samples, of which $k$ fall in $(0,a)$, then the likelihood is

$$L = \left(p\, (A-B) +B\right)^k \left((1-p)B\right)^{n-k}$$

The derivative of the log-likelihood is then and $$\frac{k (A-B)}{p\, (A-B) +B} - \frac{(n-k)B}{(1-p)B}$$

which gives me

$$p_0=\frac{k}{n}-\left(1-\frac{k}{n}\right)\frac{a}{b-a}=\frac{1}{b-a}\left(\frac{k}{n}b-a\right)$$

Not however that this is just the critical point, which does not necessarily coincide with the ML estimator. ACtually, that is valid only if $k/n > a/b$, and

$$\hat p_{ML}= \begin{cases} \begin{array}{lc} \frac{1}{b-a}\left(\frac{k}{n}b-a\right) & k \ge n \frac{a}{b}\\ 0 & elsewhere \end{array}\\ \end{cases}$$

It doesn't seem to be consistent, as, when $n\to \infty$ $k/n \to p$ (in prob.), hence $\hat p_{ML}$ won't converge to $p$. Edited: sorry for the errors; actually $n/k \to p+(1-p)a/b = [p(b-a)+a]/b$ and hence the estimator should be at least asymtotically unbiased.

-
Thanks for the input. Is there any more agreeable way to check for efficiency than to actually compute the fisher information and compare it's inverse to $Var(\hat p)$? –  Amarus Jun 13 '13 at 1:48
I'd say that this estimator is biased (even asymtotically), so it cannot be efficient (actually, the Cramer Rao bound applies only to unbiased estimators –  leonbloy Jun 13 '13 at 2:20
BTW, you might have better answers at stats.stackexchange.com –  leonbloy Jun 13 '13 at 2:20
Why would $\frac{k}{n} \to p$ for $n \to \infty$? Values from $U[0,b]$ have lie within $[0,a]$ with a probability of $\frac{a}{b}$, i.e. $\mathbb{P}(X \leq a) = p + (1-p)\frac{a}{b}$ and thus $\frac{k}{n} \to p + (1-p)\frac{a}{b}$, no? –  fgp Jun 13 '13 at 8:38
I don't understand why biasedness is a problem. We introduced the CR bound in a general form that makes no use of unbiasedness: $$Var_\theta(T) \ge (\partial/\partial \theta E_\theta(T))^2 / I(\theta),$$ where $I(\theta)$ is the Fisher information. The usual CR bound follows from this in the unbiased case. With this, we can define efficiency independent of unbiasedness, I guess? –  Amarus Jun 13 '13 at 9:06