# Statistical inference, estimation, conceptual trouble

I've just begun learning about statistical inference and I'm having a bit of trouble understanding the concepts at hand. The exercises I've done and lectures I've read kind of gloss over the details and I'm finding myself confusing various notions and although I'm able to solve the exercises from my book, I feel like I'm just brute forcing my way through it without really getting it. Below I will write my understanding of estimators:

Suppose we have some set of quantities $\theta=(\theta_1, ..., \theta_p)$ that describe some sort of system we are working with. We don't know these quantities, but we want to determine them as precisely as we can by doing measurements to the system. So we make $n$ measurements: $D_n = (X_1,...,X_n)$ - we treat each measurement as a separate random variable. We wish to produce a new random variable (the estimator) $\hat{\theta}_n=f(D_n)$ which depends on our measurements, and whose distribution gives us the best possible idea of the true value of $\theta$.

Example: We have an unknown probability distribution, $g$. We wish to find its expected value, $\theta$, by making measurements, that is, producing numbers from this distribution. Thus, we produce $n$ measurements: $D_n = (X_1, ...,X_n)$, i.e. random variables distributed according to $g$.

By its definition, $\theta = E[X_i]$ for any $1\le i\le n$.

We propose an estimator $\hat{\theta}_n=\frac{1}{n}\sum_{i=1}^{n}X_i$, which is a natural choice since by the weak law of large numbers, this converges to the expected value as $n$ goes to infinity.

The problem I have with this is that this particular estimator is conjured out of thin air, and although of course it makes sense, how do I know if it's the best one (is it?) - from what I've seen there's never any actual derivation of any estimator in my notes, they're just stated as fact - is it because it's beyond the scope of an introductory statistics course?

Related, here's an exercise I'm having a problem with:

We measure $n$ times (independently) the strength of a pure signal and then the strength of a noisy signal, yielding: $D^{(1)} _n = (X_1,...,X_n)$ (for the pure signal) and $D^{(2)}_n = (X_1 + W_1,...,X_n+W_n)$ (for the noisy signal. We use an estimator for the variance (this one too appears out of thin air):

$$\hat{s^2} = \frac{1}{n-1} \sum_{i=1}^{n} (T_i-\hat{\mu})^2$$

yielding $s^2 _1 = 0.02$ (for pure signal) and $s^2 _2 = 0.03$ (for noisy signal). Apart from that we estimate the expected values like in the previous example, giving $\mu_1 = 0.53$ and $\mu_2 = 0.79$.

We wish to estimate the expected value ($\mu$) and variance ($s^2$) of the noise.

Clearly $W_i = (X_i +W_i)-X_i$, so $$\hat{\mu}=\frac{1}{n}\sum W_i=\frac{1}{n}\sum (X_i + W_i)-(X_i) = \mu_2 -\mu_1$$

Similarly, $$\hat{s^2} = \frac{1}{n-1} \sum (W_i - \hat{\mu})^2 = \frac{1}{n-1} \sum ((X_i+W_i-\mu_2)-(X_i - \mu_1))^2$$ $$= \hat{s^2 _1} + \hat{s^2 _2} - \frac{2}{n-1} \sum(X_i+W_i-\mu_2)(X_i-\mu_1)$$

This is puzzling to me, is the last term in this negligible? The answer to the exercise just says to add the variances.

I would appreciate if someone could correct me on my understanding of this topic and point out any errors I've made.

You have asked a lot of questions. I am going to focus, very briefly, on one aspect: How do we decide which estimator to choose?

Note that in order to estimate $\theta$, we can choose any arbitrary function of the observed variables: $X_1,...X_n$. For example, we could set $\hat{\theta}=X_1$ but that is not 'efficient' in some sense as we are not using all the information we have. Clearly, some estimators are 'better' than others. Listed below are some properties that estimators should ideally have:

1. Unbiased
2. Consistent
3. Efficiency

It is not always possible to find an estimator that meets all the above conditions.

In your example of the normal distribution, it turns out that the sample mean is an unbiased, consistent and efficient estimator of the true mean.

• I see. Is there a procedure that would allow one to consistently produce such estimators? And what about the exercise - I don't understand what I'm even supposed to give as my answer - the estimator, which is a random variable, or some sort of estimate - a number? How do I produce an estimate given an estimator? Commented Jun 3, 2013 at 16:31
• There are a family of methods that are usually used: Ordinary least squares, maximum likelihood, bayesian estimation, method of moments and so on. I am guessing that you will learn about these methods and their properties later on (if at all). I will take a look at your exercise when I have more time. Perhaps, someone else will help you out in the meantime. Commented Jun 3, 2013 at 16:44