$E[X_i]=\mu$ Statistics and Probability Mysteries I realize I have over-flooded this forum with questions regarding probability recently, but I do not have access to a teacher, and have to learn these things by myself. I have made some good progress compared to what I knew about probability some weeks ago BUT, I am still very unsure of myself, and ...
I have to say I found it difficult to find reliable sources on the topic, and text books or documents on the internet explaining things really clearly. For example, while learning (reading) about sampling distribution and CLT, I wanted to understand why $E[\bar X] = \mu$.
I think I get it right here!
So I understand this from all the books I read on the topic. You have a population (lets say finite) and you have lots elements in it which you call $x_1, x_2, ... x_N$ where $N$ is the total number of elements in the population (since the population is finite). Good.
Now you can compute the population mean as: $\mu = {1\over N}\sum_{i=1}^N x_i$
The expected value equation is:
$E[X] = \sum_{i=1} {p_i X_i}$
Imagine you have 10 cards labelled with numbers on them (0, 1, or 2), and that you have 3 0s, 5 1s, and 2 2s. Now you could compute the mean of this population by doing:
$\mu = {1 \over 10}(0+0+1+1+1+1+1+2+2+2) = 1.1$$
and you could use the expected value formulation, knowing that assuming you know your population distribution before hand, you can say that your sample space is S={0,1,2} (is that correct, your sample space are the possible outcomes?). So the probability distribution would be Pr(0) = 0.2, Pr(1) = 0.5 and Pr(2) = 0.3. Thus:
E[X] = 0.2 * 0 + 0.5 * 1 + 0.3 * 2 = 1.1$$
Great we get the same results.
Now we don't want (because the population is too large or even infinite) to compute the population mean so we use a sample mean instead. We draw elements from the population, for instance $n$ of them and compute their average:
$\bar X = {1 \over n} \sum_{i=1}^n X_i$
I think I am confused here because we use $X_i$ on the right inside instead of $x$. I understand it this way: imagine you randomly draw a number between 1 and 10, imagine it is r = 3, and this number indicates the card you need to pick from the population of 10 cards (you shuffled the cards before and put them on the table from left to right). The number on this card is 2. So what $X_i$ is in the above equation for the sample mean is this number 2. Correct? This is our random variable which we could write $X_1 = 2$ (we put the card back on the table, draw with replacement, shuffle the cards again, etc.) and we repeat this experience $n$ times to get the following series of Xs: $X_1 = 2, X_2 = 0, X_3 = 1, ... X_n = 2$. And finally we compute our sample mean which is an average of all these Xs:
$\bar X = {1 \over n} (2 + 0 + 1 + ... 2)$
where the $1 \over n$ is there because again the way we drew the cards from the population was done in a "uniform" way (we didn't "give more importance" to cards on the left for instance when we drew them).
So back to our problem, proving that $E[\bar X] = \mu$. 
I think I get it wrong here!
Unfortunately I am lost here! So if I understand the equation for the expected mean correctly it's a sum of weighted random variables (where the weight is the probability associated with this random variable). So technically if we wanted to calculated the expected value of a sample mean (where the sample mean is already a mean of drawn elements from the population), we would need to calculate many samples means, weight them with their associated probability (which I understand in this case is close to a normal distribution) and add up the results. No ?
$E[\bar X] = \sum_{i=1}^{NN} w_i \bar X_i$
where $\bar X_1 = {1 \over n} \sum_{i=1}^n X_i$, $\bar X_2 = {1 \over n} \sum_{i=1}^n X_i$, etc.
So I think I am thinking wrong at this point because in books they define this expected value differently, they say since:
$\bar X = {1 \over n} \sum_{i=1}^n X_i$
thus
$E[\bar X] = E[{1 \over n} \sum_{i=1}^n X_i]$
$E[\bar X] = {1 \over n}E[X_1] + {1 \over n}E[X_2] + ... {1 \over n}E[X_n]$
and then they say something incredible in the sense to me not justified at all, which is that "since the expected value of a random value is the mean" (WHERE IS THAT PROVED?) 
EDIT: so I realise that I wrote above that $E[X]$ and the $\mu$ with my card example were the same, the problem I have with that, is that my example was only working with discrete r.v. and finite population. I believe to hold true, the $E[\bar X] = \mu$ equality should be proved for any population (finite or not) and a r.v that is either continuous or discrete. Does such a proof exist?
then we have:
$E[\bar X] = n \times {1 \over n} \mu = \mu$
So where I would really like to get some help, is really on the last part.
1) why do we compute the expected value of the mean as $E[\bar X] = E[{1 \over n} \sum_{i=1}^n X_i]$ rather than with the equation for the expected value $\sum_{i=1} {p_i X_i}$?
2) where is this coming from? "Since $X_1, X_2, ... X_n$ are each random variable, their expected value will be equal to the probability mean $\mu$."?????
I know the post is very long, but I hope this will be useful to many more people in the future, trying to understand why we use x rather than X, etc., etc. But if the real mathematicians out there, could please help me doing the last miles on my journey to "starting to understand" statistics it would be FANTASTIC!
Many thanks in advance.
 A: A "population" or a "sample" is a collection of random variables, for which we usually assume that each has the same characteristics, like mean and variance, or even that they are fully identically distributed. This is part of the initial assumptions. This means that by assumption
$E[X_1] = E[X_2] = ... = E[X_n] = \mu$
Also, yes your sample space in your example is $S=\{0,1,2\}$ i.e. the set of elementary events. Just don't confuse it with the sigma-algebras that can be defined based on this set, sigma algebras on which probabilities are actually defined, and permit to consider more complex situations than "draw one card". For example, with the same cards you may want to study the probabilities related to "draw two cards". In that case you have to consider the sigma-algebra that accomodates these "two-card" events, which are not "elementary" anymore.
A: Please allow me to add a new answer to explain for detailed.
The problem is that: Each experiment is to draw $n$ cards independently and the result is sample average $Y=\frac{1}{n}\sum_{i=1}^n X_i$. We want to compute the expectation of sample average $E[Y]$. By definition,
$$E[Y]=E[\frac{1}{n}\sum_{i=1}^n X_i]=\frac{1}{n}\sum_{i=1}^nE[X_i]=E[X_i]$$
As for why don't we use 
$$E[Y]=\sum_i p_iX_i=\sum_i x_iP(X_i=x_i)$$
is now easy to see, since the correct formula shall be
$$E[Y]=\sum_i p_iY_i=\sum_i y_iP(Y_i=y_i)$$
A: *

*Actually the expectation computed in two ways produce the same result since
$$\sum_{i=1}p_iX_i=\sum_{i=1}\frac{n_i}{n}X_i=\frac{1}{n}(\sum_{i_1=1}^{n_1}X_1+\sum_{i_2=1}^{n_1}X_2+...+\sum_{i_1=1}^{n_k}X_k)=\frac{1}{n}\sum_{i=1}X_i$$

*I think you may mistake $X_i$ as a sample with fixed value. But you'd better regard it as only a sample with undeterministic value, a random variable. So each $X_i$ has a mean and it can be computed by using $\sum_{i=1}p_iO_i$ where $O_i$ is possible values.
