sample mean, without replacement Going through a stats course and having some difficulties with point estimation. It makes sense in basic examples but when introducing "without replacement" I'm getting stuck.
A box has 4 numbered balls, 1,2,3 and 4.
A sample of 2 is drawn without replacement. What is the expected sample mean?
For 1 ball its $\frac{1}{4} \sum X_i$  which is $2.5$. Simple.
Once one ball is extracted, the probabilities change for the others, but as I don't know which ball number was removed, I can't use the same formula.
Would really appreciate some advice and pointers. Thanks.
 A: There are $\binom{4}{2} = 6$ ways to pick 2 balls from 4. Each pair thus has a $\frac{1}{6}$ probability of being picked. The possible averages are $\frac{1+2}{2}, \frac{1+3}{2},\frac{1+4}{2}, \frac{2+3}{2},\frac{2+4}{2},\frac{3+4}{2}$. Denoting the first ball by $X_1$ and the second by $X_2$, by the definition of expectation,
\begin{align*}
E[\frac{X_1+X_2}{2}] = \frac{1}{6} (3/2 + 4/2 + 5/2 + 5/2 + 6/2 + 7/2)
\end{align*}
A: The expected sample mean will always be the actual mean (here, $2.5$). This is because the expected value of the first ball is $2.5$ and the expected value of the second ball (if you don't know what the first ball was) is also $2.5$. It is always true that $E[X_1+X_2]=E[X_1]+E[X_2]$, even when $X_1$ and $X_2$ are not independent, and so $E[\frac12(X_1+X_2)]=\frac12(E[X_1]+E[X_2])$. This works for any sample size.
A: Following Especially Lime's answer, one can show that the mean of a sample without replacement from a finite population is indeed an unbiased estimate of the population mean:
If $x_1, x_2, \dots x_N$ is the population and $X_1, X_2, \dots X_n$ is a sample without replacement where $n \leq N$, then
$$
E\left[ \frac{1}{n} \sum^n_{i=1} X_i  \right] = \mu
$$
where $\mu =\frac{1}{N} \sum^N_{i=1}x_i$. Proof:
Let $1_{\lbrace x_i \in S \rbrace}$ be a random indicator variable indicating whether $x_i$ is in the sample $S$ of size $n$. We have that
$$
\sum_{i=1}^{n} X_i = \sum_{i = 1}^N x_i 1_{\lbrace x_i \in S \rbrace}
$$
Since $n$ and $N$ are fixed and every sample is equally likely, we know that 
$$
E[1_{\lbrace x_i \in S \rbrace}] = \frac{n}{N}.
$$
Therefore,
$$
E \left[\frac{1}{n} \sum_{i=1}^{n} X_i  \right] = E \left[\frac{1}{n}  \sum_{i = 1}^N x_i 1_{\lbrace x_i \in S \rbrace} \right] = \frac{1}{n}  \sum_{i = 1}^N x_i E[1_{\lbrace x_i \in S \rbrace}] = \mu
$$
