2
$\begingroup$

I was learning confidence intervals progressing slowly with few hiccups 1, 2, and wrapping up while found few more issues, one of which I have detailed here. Requesting your kind help.

I created a list containing 1s and 0s, 1 representing yellow ball and 0 otherwise, with yellow ball proportion being 60%. Then I randomly sample from this population (sample size n), take mean value and plot. I do this N no of times, so eventually I get a sampling distribution which is normal.Then I proceed to calculate Confidence Interval (CI) and see how it fares (CI calculated for 95%, so checking if CI has pop.mean 95% of the time)

Now I have multiple variables to play with. Population size T, No of experiments N, sample size n, and every time I sample, should I replace the sample back in to population or not. And then while calculating CI, to try Z or t, population or biased or unbiased SD etc. This is my premise for my first test.

Environment:
1. Population size T, fixed
2. Sample size n, varied
3. Experiment size N, varied

Applied methods:
1. Z distribution and population SD
2. Z distribution and unbiased sample SD
3. Z distribution and biased sample SD
4. T distribution and population SD
5. T distribution and unbiased sample SD
6. T distribution and biased sample SD

With above setup, I conducted test in 2 phases.

Green dots indicate, 95% or more of the set of CIs for the respective particular N, n combination has population mean, red otherwise.

Phase 1:Sampling with replacement
Every time I sample I replace.
enter image description here

Phase 2:Sampling without replacement
Every time I sample, I do not replace. I got below result.
enter image description here

As can be seen above, strangely, sampling with replacement, does not give good CI performance. We get mixed results. But sampling without replacement performs much better as sample size increases. Why is this?

Intuitively, I thought replacement would always give better results in any case (samples become independent irrespective of sample size). Is there any underlying theory am missing that explains the weird behaviour I got or the output I got is wrong?

Please find the MWE here

Dependent file: ci_helpers.py

Update: 22nd Sept 2018 We were looking at the problem with a wrong perspective. We were wondering why Sampling with replacement was doing poorly compared to Sampling without replacement. Instead if we look at why Sampling without replacement does a far better job, we get a key (thanks to siong-thye-goh) that, in our code, for Sampling without replacement, we did not use FPC (Finite Population Correction) which was thus resulting in bigger variance, so wider CIs. Once FPC introduced, both Sampling with replacement and without are behaving poorly(?!)

I am closing this and creating another question as the narrative is now changed: Why we get such a poor performance when we do not know population mean, or is that the poor performance the result of using sample SDs in each CIs?

Special thanks to Quinto whose answers were giving deeper insights in to the problem, and is still with me investigating the new issue at hand.

$\endgroup$
  • $\begingroup$ Maybe something that may help for intuition. From what I understand, your population is fixed, so taking samples without replacement correspond to taking a sample of $n$ independent random variables whereas taking samples with replacement correspond to taking a sample of $k\leq n$ independent random variables and repeating some. The thing is that most method of statistics make assumption that your data are independent which is not the case here. $\endgroup$ – P. Quinton Sep 13 '18 at 6:28
  • $\begingroup$ sampling with replacement automatically makes all samples independent?! $\endgroup$ – Parthiban Rajendran Sep 13 '18 at 6:39
  • $\begingroup$ sampling with replacement automatically makes all samples independent, so that should have worked. sampling without replacement, you can note at least for sample size less than 10% of population it should have worked as a general rule. My population size was 4000, so at least n below 400, CIs should have worked good. $\endgroup$ – Parthiban Rajendran Sep 13 '18 at 6:49
  • $\begingroup$ maybe my answer can help you get more what I meant. Let's take an extreme case, if you have a population size of $2$, then you have $2$ independent samples. If you take $2$ without replacement then you have exactly $2$ independent samples. If you take $2$ with replacement, you have a probability $1/2$ of having $2$ times the same sample and $1/2$ of having different ones. If you have only one sample it is worse for estimation purpose than if you have $2$ especially because the model you use make the assumption you have independent samples (but they are equal) $\endgroup$ – P. Quinton Sep 13 '18 at 7:01
  • $\begingroup$ thank you very much on trying to explain in detail.. I am yet to comprehend your bigger answer below (yet to learn markov and new to information theory), but from this extreme case, this is my inference.. When I have 2 samples, without replacement, ensures independent samples, but with replacement, has a definite probability of picking up same sample again. Still could not get how does it eventually negatively affects CI? Also, isnt there also a general rule, that if our sample size is below 10% of total popoulation, we are good to assume samples are independent? $\endgroup$ – Parthiban Rajendran Sep 13 '18 at 10:13
1
+150
$\begingroup$

Let me give a intuition through information theory. Most methods of statistics assume that the samples are independent, if this is not the case we usually try to transform our data such that it is.

Consider the two different scenarios :

  • We take a sample of $n$ independent samples $X_1$.

  • We take a sample of $k\leq n$ independent samples $Y_2$ and repeat random samples such that the amount of samples is $n$, put the $n$ samples in $X_2$.

Suppose we have a model $\mathcal{H}$ that is used to generate the samples (we suppose it is random in some fashion). An interesting measure from information theory can help you get some intuition, it is called mutual information. The mutual information correspond to the amount of information a random variable gives about another, as an example if it is $0$, then they are independent and if the information is maximal then there exists a mapping such that the mapping applied to the first one is almost surely equal to the second one (this is not actually true if I recall correctly but it gives some intuition).

What we are interested in is comparing $I(\mathcal{H}, X_1)$ and $I(\mathcal{H}, X_2)$. Observe that $\mathcal{H} - Y_2 - X_2$ form a Markov chain, that is we generate the additional samples of $X_2$ independently of $\mathcal{H}$ when we have the knowledge of $Y_2$. We can apply the Data Processing Inequality which gives that $I(\mathcal{H}, X_2) \leq I(\mathcal{H}, Y_2)$. This is very intuitive since given $Y_2$, $\mathcal{H}$ and $X_2$ are independent, so the additional randomness of $X_2$ compared to $Y_2$ is basically just noise.

Now finally observe that $I(\mathcal{H}, X_1) \geq I(\mathcal{H}, Y_2)$ since $k\leq n$. So in the end $I(\mathcal{H}, X_1) \geq I(\mathcal{H}, X_2)$ which means that $X_1$ contains more information about the model than $X_2$

$\endgroup$
  • $\begingroup$ When I say "with replacement" I meant, we take one sample, note down the value, and keep it back in the population and sample again from entire population. This way I construct sample set of n samples (which thus may have repetition but population itself is bernoulli so we dont know). So at any time, you have entire population to sample from. So $X2$ construction is not clear for me. Can you please elaborate on that? $\endgroup$ – Parthiban Rajendran Sep 13 '18 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.