4
$\begingroup$

I am self learning estimation theory and finding it quite difficult to grasp the utility of Cramer Rao Lower bound. In text books and online tutorials always say that one should derive the CRLB of the estimator. If the variance of the estimator is greater than equal to the inverse of the Fisher information, then we say that no other better estimator exists. The inverse of the Fisher information is the CRLB. If the variance is equal to the CRLB, then the estimator is efficient. Intuitively, an estimator is nothing but a formula or an expression that is used to find an unknown value/ parameter.

1) Is there a better intuitive way to explain what CRLB bound tells us and why we need it?

2) What is meant by efficient estimator and efficiency to do what? With what do we compare the efficiency of an estimator. These questions may be trivial but I found very difficult to extract key information from highly mathematical heavy stuff.

3) Do we discard the estimator if inefficient?

Please correct me if any information is wrong. Thank you.

$\endgroup$
  • 1
    $\begingroup$ It's not necessarily true that there is no estimator better than the best unbiased estimator. In some cases, some biased estimators are superior to all unbiased estimators. $\endgroup$ – Michael Hardy Oct 28 '17 at 22:41
  • $\begingroup$ Begin by looking at the 'main' answer to this question . $\endgroup$ – BruceET Oct 29 '17 at 1:43
2
$\begingroup$

Please begin by looking at the main answer (by @Learner) to this question. Then I have a simple example that may help illustrate.

Consider the problem of estimating the success probability $\theta$ of the distribution $\mathsf{Binom}(n, \theta)$ using information from $n$ trials. The maximum likelihood estimator of $\theta$ is $\hat \theta = X/n,$ where $X$ is the observed number of successes.

The PDF of $X$ is $$f(x\,|\,\theta) = {n \choose x}\theta^x(1-\theta)^{n-x},$$ for $x = 0, 1, \dots, n.$ Given $n$ and $\theta,$ the PDF allows you to calculate $P(X = x).$

Viewed as a function of $\theta$ for observed $x$ this same relationship is known as the likelihood function. $$\ell(x\,|\,\theta) = {n \choose x}\theta^x(1-\theta)^{n-x} \propto \theta^x(1-\theta)^{n-x},$$ for $0 \le \theta\le 1.$ The symbol $\propto$ ("proportional to") indicates that for observed $x$ the constant factor ${n \choose x}$ has been omitted. Many authors take the point of view that the likelihood function is defined only 'up to a positive constant'. Then the maximum likelihood estimator (MLE) of $\theta$ is found by maximizing $\ell(x\,|\,\theta)$.

Suppose $n = 5$ and $x = 2.$ Then the MLE $\hat \theta = 0.4$ is the value of $\theta$ that maximizes the curve in the left panel of following figure. Also, if $n=50$ and $x = 20,$ then again the MLE $\hat \theta = 0.4.$

But there is more 'information' in 50 observations than in five, so the likelihood curve in the right panel has a 'sharper' maximum.

enter image description here

The second derivative of the curve near $\hat \theta = 0.4$ is greater on the right. Very roughly speaking, the Fisher information gives the expected curvature.

$\endgroup$
  • $\begingroup$ Thank you for your answer and the illustration. I have 2 doubts and one request (doubt1) what is unbiased? (doubt2) MLE is nothing but an algorithm to estimate an unknown. Can the estimate be used as an objective function minimizing or maximizing of the estimate would allow some other task to occur?For instance there are heuristic optimization methods such as particle swarm, ant colony etc. These algorithm use an objective function to estimate or find out the best solution. MLE uses the likelihood function as the objective function, maximization of it yields the unknown parameter. $\endgroup$ – Srishti M Oct 29 '17 at 4:23
  • $\begingroup$ After the unknown parameter is found, can that estimate be used as an objective function for some other estimation task? This is quite unrelated to what I have posted here, but would appreciate your insights in this. Another request is that I have posted another question here math.stackexchange.com/questions/2492793/… where I don't know what is the proper notations and confused about it. Would really appreciate if you can kindly answer that as well, if possible. Thank you for your time. $\endgroup$ – Srishti M Oct 29 '17 at 4:23
  • $\begingroup$ I am not sure what additional use you want to make of Fisher information. I have never done this, but it might be worthwhile asking a specific question here or on our sister statistics site. // I will look at you other question. I hope it gets a good answer soon. $\endgroup$ – BruceET Oct 29 '17 at 15:10
2
$\begingroup$

The Cramer Rao bound is fundamental to the study of statistics, let me try to explain you why without using a lot of symbols. An estimator does exactly that: it estimates. Given the data you've seen what do you think a certain parameter really is. Now, thinking about estimators one can ask whether or not ones are better than others. Have you realized that people don't like high variance of things? There is a weird feeling when something changes a lot. The whole study of statistics follows as a principle trying to reduce variance. Wouldn't it be nice if you could find among all possible unbiased estimators of your parameter, one with minimal variance? I.e. one which looks more solid/stable? That is the result of Cramer and Rao, not only do they tell you that there is a lower bound for the variance, they also tell you how to compute it.

$\endgroup$
  • 1
    $\begingroup$ Great answer as well, simple to follow......thank you $\endgroup$ – Srishti M Oct 29 '17 at 4:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.