Why Sampling without replacement gives better CI performance?

Question

I was learning confidence intervals progressing slowly with few hiccups 1, 2, and wrapping up while found few more issues, one of which I have detailed here. Requesting your kind help.

I created a list containing 1s and 0s, 1 representing yellow ball and 0 otherwise, with yellow ball proportion being 60%. Then I randomly sample from this population (sample size n), take mean value and plot. I do this N no of times, so eventually I get a sampling distribution which is normal.Then I proceed to calculate Confidence Interval (CI) and see how it fares (CI calculated for 95%, so checking if CI has pop.mean 95% of the time)

Now I have multiple variables to play with. Population size T, No of experiments N, sample size n, and every time I sample, should I replace the sample back in to population or not. And then while calculating CI, to try Z or t, population or biased or unbiased SD etc. This is my premise for my first test.

Environment:
1. Population size T, fixed
2. Sample size n, varied
3. Experiment size N, varied

Applied methods:
1. Z distribution and population SD
2. Z distribution and unbiased sample SD
3. Z distribution and biased sample SD
4. T distribution and population SD
5. T distribution and unbiased sample SD
6. T distribution and biased sample SD

With above setup, I conducted test in 2 phases.

Green dots indicate, 95% or more of the set of CIs for the respective particular N, n combination has population mean, red otherwise.

Phase 1:Sampling with replacement
Every time I sample I replace.

Phase 2:Sampling without replacement
Every time I sample, I do not replace. I got below result.

As can be seen above, strangely, sampling with replacement, does not give good CI performance. We get mixed results. But sampling without replacement performs much better as sample size increases. Why is this?

Intuitively, I thought replacement would always give better results in any case (samples become independent irrespective of sample size). Is there any underlying theory am missing that explains the weird behaviour I got or the output I got is wrong?

Please find the MWE here

Dependent file: ci_helpers.py

Update: 22nd Sept 2018 We were looking at the problem with a wrong perspective. We were wondering why Sampling with replacement was doing poorly compared to Sampling without replacement. Instead if we look at why Sampling without replacement does a far better job, we get a key (thanks to siong-thye-goh) that, in our code, for Sampling without replacement, we did not use FPC (Finite Population Correction) which was thus resulting in bigger variance, so wider CIs. Once FPC introduced, both Sampling with replacement and without are behaving poorly(?!)

I am closing this and creating another question as the narrative is now changed: Why we get such a poor performance when we do not know population mean, or is that the poor performance the result of using sample SDs in each CIs?

Special thanks to Quinto whose answers were giving deeper insights in to the problem, and is still with me investigating the new issue at hand.

Answer 1

Let me give a intuition through information theory. Most methods of statistics assume that the samples are independent, if this is not the case we usually try to transform our data such that it is.

Consider the two different scenarios :

• We take a sample of $n$ independent samples $X_1$.

• We take a sample of $k\leq n$ independent samples $Y_2$ and repeat random samples such that the amount of samples is $n$, put the $n$ samples in $X_2$.

Suppose we have a model $\mathcal{H}$ that is used to generate the samples (we suppose it is random in some fashion). An interesting measure from information theory can help you get some intuition, it is called mutual information. The mutual information correspond to the amount of information a random variable gives about another, as an example if it is $0$, then they are independent and if the information is maximal then there exists a mapping such that the mapping applied to the first one is almost surely equal to the second one (this is not actually true if I recall correctly but it gives some intuition).

What we are interested in is comparing $I(\mathcal{H}, X_1)$ and $I(\mathcal{H}, X_2)$. Observe that $\mathcal{H} - Y_2 - X_2$ form a Markov chain, that is we generate the additional samples of $X_2$ independently of $\mathcal{H}$ when we have the knowledge of $Y_2$. We can apply the Data Processing Inequality which gives that $I(\mathcal{H}, X_2) \leq I(\mathcal{H}, Y_2)$. This is very intuitive since given $Y_2$, $\mathcal{H}$ and $X_2$ are independent, so the additional randomness of $X_2$ compared to $Y_2$ is basically just noise.

Now finally observe that $I(\mathcal{H}, X_1) \geq I(\mathcal{H}, Y_2)$ since $k\leq n$. So in the end $I(\mathcal{H}, X_1) \geq I(\mathcal{H}, X_2)$ which means that $X_1$ contains more information about the model than $X_2$