# Why is an unbiased estimator supposed to give a better approximation

I am learning stats myself, I learned about unbiased estimators yesterday. This is the definition of unbiased estimators:

Let $\theta$ be a fixed parameter for a probability distribution $P$ and let $X_1,X_2,\ldots, X_n$ be samples drawn from $P$, then a statistic $u(X_1,X_2,\ldots,X_n)$ is called an unbiased estimator for $\theta$ if $E(u(X_1,X_2,\ldots,X_n)=\theta$

These are some of my confusions:

1) Is this definition complete and correct?

2) The condition that the expectation of statistic should equal the parameter seems completely arbitrary to me. I mean to ask what does this imply and why is it a desirable property?

For example, supposed I have $n$ iid samples $x_1,x_2,\ldots,x_n$ ($x_i$ are real numbers) from an unknown distribution $P$. I want to estimate a fixed parameter $m$ (like median/variance etc) of the distribution $P$ and I have a statistic whose expectation equals $m$. Why should I expect $u(x_1,x_2,\ldots,x_n)$ to approximate $\theta$ and how close is this approximation supposed to be?

• If you throw a die a 1000 times then is there any reason to believe that statistic $u=\frac1{1000}(x_1+\cdots+x_{1000})$ will approximate $3.5$ (its mean)? Well, the law of large numbers for instance. It sounds as if you see that as irrational thinking ("...completely arbitrary..."). The definition of unbiased estimator is okay. Questions like "how close" are part of the theory and cannot be answered in a general sense. – drhab Dec 21 '17 at 14:30
• @drhab Okay that makes sense for this special instance. What about a general parameter and a general statistic whose expectation equals the parameter? – ELLANHER Dec 21 '17 at 14:33
• I think that - if you go on with learning stats - you will engage lots of examples that will make your confusion/disbelief melt away. Yesterday you learned about unbiased estimators so you are at the beginning of a long journey (good luck with it). – drhab Dec 21 '17 at 14:43
• I am not sure if examples would melt away my disbelief. I am more interested in finding out a general proof. I don't have to completely understand it, I just want to make sure that one exists. – ELLANHER Dec 21 '17 at 14:47

Firstly, you are using the word "sample" incorrectly. The values of $X_1,\ldots,X_n$ are not samples; they are observations in a sample whose size is $n.$

Your definition of "unbiased" is incomplete. The definition should say that for EVERY value of the parameter $\theta,$ this equality holds. As $\theta$ changes, then so does the expected value, and the definition should say they always remain equal.

As as been pointed out, in some cases some biased estimators perform better in the mean-squared-error sense than does the best unbiased estimator. And example is that the biased MLE for the variance of a normally distributed population has a uniformly smaller mean squared error than does the best unbiased estimator. And for the mean of an $n$-variate normal distribution, the MLE is unbiased but does not perform as well as some biased Bayesian estimators.

And sometimes unbiasedness is a very bad thing. This paper gives an example.

• If the mean squared error of an estimator is low, that means its variance is lower which implies there is a high probability of a sample drawn from to lie close to its expected value. Is my reasoning valid? – ELLANHER Dec 22 '17 at 3:49
• @Ali : Yes. $\qquad$ – Michael Hardy Dec 22 '17 at 5:41
• Okay so MSE is a better metric for assessing the quality of an estimator than bias alone? – ELLANHER Dec 22 '17 at 6:19
• Also a last question, does law of law of large numbers has anything to do with this? – ELLANHER Dec 22 '17 at 6:25
• Let $T$ be an estimator of $\tau.$ Then $\text{MSE}_\tau = E(T-\tau)^2 = Var(T) + [B(T)]^2.$ where $B(T) = E(T) - \tau$ is the bias. // Re link to interesting paper: In estimating normal $\sigma^2$ I seem to recall dividing by $n+1$ gives smaller MSE than dividing by $n$ or $n-1.$ – BruceET Dec 22 '17 at 9:05