I am struggeling to understand the how an estimator is arrived and whether it can be determined it is biased or not.

I have this example Let $X_1 , X_2 ,\ldots, X_7$ denote a random sample from a population having mean $\mu$ and variance $\sigma^2$. Consider the following estimator of $\mu$: $$\hat{\Theta}_1 = \frac{X_1 + X_2 +\cdots+ X_7}{7}$$ $$\hat{\Theta}_2 = \frac{2X_1 − X_6 + X_4}{2}$$

a) Is either Estimator unbiased?

So since these are trying to estimate the $\mu$, $E[\hat{\Theta}] = \mu$. So i've done it like this $E[\hat{\Theta}_1] = \frac{E[X_1] + E[X_2]+\cdots+E[X_7]}{7}$. And since the expected value has been taken from the same population, we would expect it would be the same in all cases therefore it can be rewritten as $E[\hat{\Theta}_1] = \frac{7E[X]}{7} = E[X]$. which lead to showing that it is unbiased.

My confusing occurs when i use the Expected value on an estimator estimating $\mu$, it does not make sense to do that, (and seems incorrect to say so), since it's like finding the expected value of $\mu$, which obviously is $\mu$. So what am I doing wrong?


"What am I doing wrong?" Nothing here. Your approach is correct: $\hat{\theta}_1$ is an unbiased estimator, for the reason you give (basically, it has to do with linearity of expectation). When $X_1,\dots,X_n$ are identically distributed and have finite expectation $\mu$, $$ \mathbb{E}\!\left[\frac{1}{n}\sum_{k=1}^n X_k\right] = \frac{1}{n}\sum_{k=1}^n \mathbb{E}[X_k] = \frac{1}{n}\sum_{k=1}^n \mu = \mu $$

For the same reason, $\hat{\theta}_2$ is unbiased as well: $$ \mathbb{E}\!\left[\hat{\theta}_2\right] = \frac{2\mathbb{E}[X_1] - \mathbb{E}[X_6]+\mathbb{E}[X_4]}{2} = \frac{2\mu - \mu+\mu}{2} = \mu\ . $$

Note that while both are unbiased estimators, the first one may be better, as it will have smaller variance... i.e., while both expectations are the desired value $\mu$, the second will be much less accurate (more subject to random fluctuations).

-- Attempt at addressing the larger question: assume the goal is to estimate a quantity $\theta$ of the distribution, assuming you have access to identically distributed (and, most of the time, assumed to be independent) r.v.s/samples from it, $X_1,X_2,\dots$. You basically want to come up with a nice function $f$ or sequence of functions $(f_n)$ such that either

  • $f(X_1,\dots, X_k)$ is with high probability close to $\theta$, or
  • $f_n(X_1,\dots,X_n)$ converges (when $n$ grows) towards $\theta$, in some specific sense.

Here, requiring an unbiased estimate you also want $\mathbb{E}[f_n(X_1,\dots, X_n)] = \theta$ (for all $n$), when the probability is over the realizations of $X_1,X_2,\dots$. Basically, your estimator is also a random variable, which depends on the $X_i$'s (being a function of them); and you can as such compute its expectation, variance, etc.

Now, for the case of estimating the expected value, things can seem a bit circular, since your estimator is basically a linear function of the $X_i$'s, and computing its expected value amounts to compute their expected value, which is what you are trying to estimate. But it does make sense nonetheless (it's not because it looks to simple to be true that it's not true); for instance, one could come up with other estimators for the expected value, say for instance $ Y = \ln \frac{e^{X_1}+e^{X_2}+\dots+e^{X_n}}{n}. $ Computing the expected value of this new random variable will not be as straightforward; yet eventually it will also boil down to using the fact that the expectation of each $X_i$ is $\mu$ (as well, maybe, as other assumptions like independence of the $X_i$'s).

| cite | improve this answer | |
  • $\begingroup$ "My confusing occurs when i use the Expected value on an estimator estimating µ, it does not make sense to do that, (and seems incorrect to say so), since it's like finding the expected value of µ, which obviously is µ. So what am I doing wrong?" $\endgroup$ – Did Dec 28 '14 at 16:08
  • $\begingroup$ I actually didn't quite understand what the OP meant by this. $\endgroup$ – Clement C. Dec 28 '14 at 16:09
  • $\begingroup$ You may not have but this is the OP's question, isn't it? $\endgroup$ – Did Dec 28 '14 at 16:10
  • $\begingroup$ It seems incorrect to say so , it's like saying what do expect from a estimator estimating µ => which is obviously µ... it doesn't seem like a proof but more like a "no shit sherlock" - moment (and not statical moments..)/stating the obvious $\endgroup$ – sas Dec 28 '14 at 16:12
  • $\begingroup$ @ClementC. You see? This question is not about checking that $E(\hat\theta)=\theta$ for various $\hat\theta$ (the OP seems to know how to do that), but about the statistical approach to estimation of a parameter of the model, per se. $\endgroup$ – Did Dec 28 '14 at 16:15

You wrote: "since these are trying to estimate the $\mu$, $E[\hat{\Theta}] = \mu$."

That's your mistake. One cannot conclude that the expected value of an estimator is $\mu$ merely because $\mu$ is what one is trying to estimate. Rather, that is exactly the thing to be proved.

For example, if $X_1,\ldots,X_n\sim\mathrm{i.i.d.}\ N(\mu,\sigma^2)$ then the maximum-likelihood estimator of $\sigma^2$ is biased: its expected value is not $\sigma^2$ but $(n-1)\sigma^2/n$.

Your way of proving that the expected value of the estimator is $\mu$ is correct.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.