# How do we know wheter particular statistic is biased or unbiased?

These are definitions I know (correct me if I'm wrong):

• statistic - a function of a random sample $$x_1, x_2, ... x_n$$, i.e. mean
• estimator - statistic chosen to serve as an estimate of some parameter of a model, i.e. we assume that our sample comes from an exponential distribution with corresponding probability mass function $$f(x_1, ... x_n;\theta) = \theta e^{-\theta x}$$. We don't know the real value of $$\theta$$ so we use aritmetic mean of the sample to estimate it and imagine how the real distribution of variable $$X$$ looks like. In this case the mean is our estimator of $$\theta$$.
• unbiased estimator - an estimator such that it's expected value is equal the REAL VALUE of estimated parameter: $$E[\tilde{\theta}] = \theta_0$$

My question is: how could we ever know the real value of $$\theta$$? If we knew it why would we try to estimate? Maybe is it proven experimentally which estimators are biased and which aren't for all known estimators or sth like that???

You don't know the real value of $$\theta$$, but you can simply define $$\theta = \mathbb{E}\left(X\right)$$ for a generic observation $$X$$. In other words, $$\theta$$ is simply notation for the mean that you want to estimate. Then, as long as each observation has the same mean (that is an assumption you have to make), the sample mean will be an unbiased estimator for any possible value of $$\theta$$. That can be proved analytically; you do not need to "verify" it in practice, but the purpose of the result is to show you that the sample mean is in some sense the "right" estimator to use in applications.

Unbiasedness is derived from the mathematical structure of the estimator. For example, if you consider the sample standard deviation

$$\hat{S}^2 = \frac{1}{n-1}\sum^n_{i=1} \left(X_i - \bar{X}\right)^2,$$

where you assume that $$X_i$$ are independent with the same mean $$\mu$$ and variance $$\sigma^2$$, you can analytically compute $$\mathbb{E}\left(\hat{S}^2\right)$$ and find that it is equal to $$\sigma^2$$, regardless of what $$\sigma^2$$ actually is. If you replace $$\frac{1}{n-1}$$ by $$\frac{1}{n}$$ out in front, this will no longer be true -- for any value of $$\sigma^2$$, the expected value of such an estimator will always be slightly smaller than $$\sigma^2$$ for any finite $$n$$. This tells you that using $$\frac{1}{n-1}$$ is in some sense the "correct" way to construct an estimator of the variance, so that is the estimator you should use in practice.