Indicators and sampling with/without replacement

Question

I was reading the problem in this post: Solving for an expected value from discrete random variables:

An urn contains $30$ marbles of which 8 are black, $12$ are red, and $10$ are blue. Randomly, select four marbles without replacement. Let $X$ be the number of black marbles in the sample of four. Compute $E(X)$.

I understand the solution using indicator random variables:

We consider that the drawn marbles can be placed in a line, and on doing so let $X_i$ be the indicator that the $i$-th marble in this line is black.   The probability that a particular marble in the line will be black is: $8/30$.   Thus $\mathsf E(X_i)=8/30$ for all four marbles.

By the Linearity of Expectation then: $E(X) $ $= \sum\limits_{i=1}^4 E(X_i) = \frac{16}{15}$

By the same logic, if we sampled with replacement, we should get the same answer. To me this is unintuitive. For instance, if we only had two black balls and we ask the same problem, wouldn't the two $E(X)$ still be the same? If we are sampling without replacement shouldn't the fact that sampling $3$ or $4$ black balls being impossible make this value different from where you sample with replacement (and it is possible to sample $3$ or $4$ balls)?

Can someone give me some intuition of why this is reasonable? Much appreciated!

Answer 1

• The probability the first sampled marble is black is $\frac{8}{30}=\frac4{15}$.

• Similarly the probability the second sampled marble is black is also $\frac4{15}$, whether the sample is with or without replacement, due to symmetry/exchangeability among the balls.

• Also for the third and for the fourth.

• By linearity of expectation, this makes the expected number of black marbles in the sample of four marbles $\frac4{15}+\frac4{15}+\frac4{15}+\frac4{15}=\frac{16}{15}$ whether the sample is with or without replacement.

In general, linearity of expectation does not need independence, which here comes with sampling with replacement. The variances would be different: since there is negative correlation between the balls when sampling without replacement, the variance of number of black marbles in the sample of four would be smaller without replacement than with replacement. To take a more extreme example, sampling $30$ marbles, the expectation without replacement is going to be $8$ with zero variance, while with replacement is going to be $30\times \frac{4}{15}=8$ with a variance of $\frac{88}{15}$ .

Answer 2

By the same logic, if we sampled with replacement, we should get the same answer. To me this is unintuitive. For instance, if we only had two black balls and we ask the same problem, wouldn't the two $\mathsf E(X)$ still be the same?

Many do find the Linearity of Expectation counterintuitive. However, it works. Let us try drawing without replacement from a population of $2$ black and $28$ non-black.

$\qquad\begin{align}\mathsf E(X) &=\mathsf P(X=1)+2~\mathsf P(X=2)\\&=\dfrac{\dbinom 21\dbinom{28}{3}}{\dbinom{30}4}+\dfrac{2\dbinom 22\dbinom{28}{2}}{\dbinom{30}4}\\&=\dfrac{4}{15}\\[2ex]\mathsf E\left(\sum_{i=1}^4X_i\right) &= \mathsf E(X_1)+\mathsf E(X_2)+\mathsf E(X_3)+\mathsf E(X_4) \\&=4~\mathsf P(X_1=1)\\&=\dfrac{4\cdot 2}{30}\\&=\dfrac{4}{15}\end{align}$

Answer 3

If draw with replacement, then each draw is independent and identical Bernoulli experiment, $$P(X_i=black)=p=\frac{8}{30}, ~~i=1,2,3,4$$

Let $X=X_1+...+X_4$, then $X$ follows binomial distribution.

$$E(X)=np=\frac{32}{30}$$

Answer 4

The reason this initially seems counterintuitive (to me, at least) is that when I consider the drawing-without-replacement scenario, the first thought that shoots through my head (again, could just be me) is that if a previously-drawn ball was black then the probability of subsequently-drawn balls also being black is reduced. However, what takes a bit longer to occur to me is that the reverse is also true: if a previously-drawn ball was not black, the probability of subsequently-drawn balls being black is increased. So my rudimentary and completely intuitive interpretation of what linearity of expectation says in this particular scenario is that the above two cases "even out" and so the expected number of black balls will be the same as if the balls were being replaced.