13
$\begingroup$

If some maximum likelihood estimator (MLE) turns out to be unbiased (which does not necessarily holds), then does it achieve the Cramer-Rao lower bound (CRLB) even in finite sample? (It does when the parameter to estimate is the mean of some normal, or Poisson, or binomial distribution, for example.)

I feel there should be some MLE which is unbiased but does not achieve CRLB, but I could not give an example. So I am wondering if the claim actually holds.

$\endgroup$
  • 3
    $\begingroup$ As we know, MLE does not necessarily achieve CRLB and MLE is not necessarily unbiased but you are asking whether MLE can fail to achieve CRLB and be unbiased. Good question, +1. $\endgroup$ – Did May 18 '14 at 7:42
  • $\begingroup$ OP: Please check that you agree with the new version of the last sentence. $\endgroup$ – Did May 18 '14 at 14:34
7
$\begingroup$

An example can be given, when we have a misspecification.
Assume that we have an i.i.d. sample of size $n$ of random variables following the Half Normal distribution. The density and moments of this distribution are

$$f_H(x) = \sqrt{2/\pi}\cdot \frac 1{v^{1/2}}\cdot \exp\big\{-\frac {x^2}{2v}\big\}\\ E_H(X) = \sqrt{2/\pi}\cdot v^{1/2}\equiv \mu_x,\;\; \operatorname{Var}_H(X) = \left(1-\frac 2{\pi}\right)v$$

The log-likelihood of the sample is

$$L(v\mid \mathbf x) = n\ln\sqrt{2/\pi}-\frac n2\ln v -\frac {1}{2v}\sum_{i=1}^nx_i^2$$

The first and second derivatives with respect to $v$ are

$$\frac {\partial}{\partial v}L(v\mid \mathbf x) = -\frac n{2v} + \frac {1}{2v^2}\sum_{i=1}^nx_i^2,\;\; \frac {\partial^2}{\partial v^2}L(v\mid \mathbf x) = \frac n{2v^2} - \frac {1}{v^3}\sum_{i=1}^nx_i^2$$

So the Fisher Information for parameter $v$ is

$$\mathcal I(v) = -E\left[\frac {\partial^2}{\partial v^2}L(v\mid \mathbf x)\right] = -\frac n{2v^2} + \frac {1}{v^3}\sum_{i=1}^nE(x_i^2) = -\frac n{2v^2} + \frac {n}{v^3}E(X^2)$$

$$=-\frac n{2v^2} + \frac {n}{v^3}\left[\operatorname{Var}(X)+\left(E[X])^2\right)\right] = -\frac n{2v^2} + \frac {n}{v^3}v$$

$$\Rightarrow \mathcal I(v) = \frac n{2v^2}$$

The Fisher Information for the mean $\mu_x$ is then

$$\mathcal I (\mu_x) = \mathcal I(v) \cdot \left(\frac {\partial \mu_x}{\partial v}\right)^{-2} = \frac n{2v^2}\cdot \left(\sqrt{2/\pi}\frac 12 v^{-1/2}\right)^{-2} = \frac {n\pi}{v}$$

and so the Cramer-Rao lower bound for the mean is

$$CRLB (\mu_x) = \left[\mathcal I (\mu_x)\right]^{-1} = \frac {v}{n\pi}$$

Assume now that we want to estimate the mean using maximum-likelihood, but we make a mistake: we assume that these random variables follow an Exponential distribution with density

$$g(x) = \frac 1{\beta}\cdot \exp\big\{-(1/\beta)x\big\}$$ The mean here is equal to $\beta$, and the maximum likelihood estimator will be

$$\hat \beta_{mMLE} = \hat E(X)_{mMLE} = \frac 1n\sum_{i=1}^nx_i$$ where the lowercase $m$ denotes that this estimator is based on a misspecified density. Nevertheless, its moments should be calculated based using the true density that the $X$'s actually follow. Then we see that this is an unbiased estimator, since

$$E_H[\hat E(X)_{mMLE}] = \frac 1n\sum_{i=1}^nE_H[x_i] = E_H(X) = \mu_x$$

while its variance is

$$\operatorname{Var}(\hat E(X)_{mMLE}) = \frac 1n\operatorname{Var}_H(X) = \frac 1n\left(1-\frac 2{\pi}\right)v$$

This variance is greater than the Cramer-Rao lower bound for the mean because

$$ \operatorname{Var}(\hat E(X)_{mMLE}) = \frac 1n\left(1-\frac 2{\pi}\right)v > CRLB (\mu_x) = \frac {v}{n\pi} $$

$$\Rightarrow 1-\frac 2{\pi} > \frac {1}{\pi} \Rightarrow 1 > \frac 3{\pi}$$

which holds. So we have an MLE which is unbiased but does not attain the Cramer-Rao lower bound for the magnitude that it estimates. Its efficiency is

$$\frac {CRLB (\mu_x)}{\operatorname{Var}(\hat E(X)_{mMLE})} = \frac {\frac {v}{n\pi}}{\frac 1n\left(1-\frac 2{\pi}\right)v} = \frac 1{\pi - 2} \approx 0.876$$

Note that the MLE for the mean under the correct specification is biased, with a downward bias.

$\endgroup$
  • $\begingroup$ I didn't think about misspecification case. Thank you very much! $\endgroup$ – atohs May 26 '14 at 20:17
1
$\begingroup$

This may be useful (adapted from "Theory of Point Estimation" 2e by Lehmann and Casella, Section 2.5 on the Information Inequality)

Assume the parameter lives in $\Omega$ which is an open interval (could be infinite), such that $P_\theta$ has common support $A$ independent of $\theta$ and $\frac{\partial p_\theta (x)}{\partial \theta}$ exists and is finite for any $x\in A$ and $\theta in \Omega$.

Theorem (similar to 2.5.12): Assume $\delta$ is an unbiased estimator of $\theta$ with finite variance under any $\theta \in \Omega$. Then, $\delta$ attains the CRLB iff there exists a cont. differentiable function $\phi(\theta)$ such that $p_\theta (x) = C(\theta) e^{\psi(\theta) \delta(x)} h(x)$ is a density (i.e. $p_\theta$ constitutes an exponential family).

There are some other similar theorems cited in the same section.


Consider a symmetric distribution supported on $(-1/2,1/2)$, $f(x)$ which is sufficiently smooth with unique peak at $0$. Then, for $\theta \in \mathbb{R}$, consider the family of distributions $p_\theta (x) = f(x - \theta)$. The Fisher information is $\int_{-1/2}^{1/2} \left( \frac{f'(x)}{f(x)}\right)^2 f(x) dx$.

Consider the one sample case. The MLE is the sample you observe, so we need to see if $var(X) > \frac{1}{\int_{-1/2}^{1/2} \left( \frac{f'(x)}{f(x)}\right)^2 f(x) dx}$.

Consider $f(x) = -6(x+1/2)*(x-1/2)$ on $(-1/2,1/2)$. The LHS is 1/20. The RHS is $0$.

$\endgroup$
  • $\begingroup$ your example is exactly what I wanted to see! Thank you for your help. $\endgroup$ – atohs May 26 '14 at 20:17
  • $\begingroup$ By the way, it seems that the support of the distribution depends on theta in this example, while CRLB theorem usually assumes that the family of distributions have the common support so that we can change integration (w.r.t. X) and differentiation (w.r.t. theta). Do you think this example is still valid? $\endgroup$ – atohs May 27 '14 at 0:19
  • $\begingroup$ Well, all you really need is the covariance inequality and some differentiability, so i think you can change $f$ to make it work. But you are right - the example isn't directly valid in the usual case since you usually make the assumptions in the first part of the answer to do the CRLB. However, the first part of the answer does characterize the case for when the equality is met under those assumptions. $\endgroup$ – Batman May 27 '14 at 13:30
  • $\begingroup$ I got it. Thanks again :) $\endgroup$ – atohs May 27 '14 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.