Understanding the unbiased estimator I am having a hard time understanding what an estimator actually is (I miss the intuition.).
The definition (for unbiased estimator) is as follows: $T$ is unbiased for the parameter $\theta$ if $E[T] = \theta$, irrespective of the value of $\theta$.
In this case, what is $\theta$? Is it the mean of the random variables that $T$ estimates? Does that mean that an unbiased estimator is more efficient than a biased estimator, since it depends on all possible random variables $(X_1,\ldots, X_n)$?
 A: Yep! That's exactly right. I'll try to answer your questions, too.
Any function $\delta(X_1, \ldots, X_n)$ of the data is AN estimator. For instance, the sample mean $\overline{X}_n$ is an estimator, as is the constant function of the data 10. Viewed this way, it is clear that unbiasedness is a special property.
Unbiased is not a synonym for good when it comes to estimators. For example, the following is an unbiased, but obviously stupid estimator:
$$
  \delta(\mathbf{X}) = \begin{cases}
  \overline{X}_n + 10^{10} \\
  \overline{X}_n - 10^{10}
  \end{cases}
$$
with probability 1/2 for each case. Much more often, it will be the case that an unbiased estimator is technically dominated (in terms of MSE) by some biased estimator.
As for what is $\theta$, it is the parameter of interest. This is certainly confusing, as one would presume that the parameter of interest is unknown, even unknowable. Nonetheless, we still know that it exists and, therefore, we can say things about it.
For example, we can say $E(\theta) = \theta$, whatever $\theta$ happens to be. This is analogous to saying $(7 \sqrt{a})^3 = 343 a^{\frac{3}{2}}$, regardless of what the true value of $a$ happens to be.
Of course, it is the job of theoretical statisticians to make this into a tool which would be more useful in an applied setting.
Finally, as a concrete example, consider, again, the sample mean. Let $X_1 , \ldots, X_n$ be IID. Then we have:
$$
E(\overline{X}_n) = E \left( \frac{1}{n} \sum_{i=1}^n X_i \right) = \frac{1}{n} \sum_{i=1}^n E(X_i) = \frac{1}{n} \sum_{i=1}^n \mu = \mu. 
$$
This proves that, no matter what the value of the ''population mean'' $\mu$, the expectation of the sample mean (but not the observed value of the sample mean) is the same as the population mean.
In other words, by definition, $\overline{X}_n$ is an unbiased estimator of $\mu$.
Does that help?
