Sampling Distributions: Sample size of 1 vs Sample Size of m

I saw this example from a website

Suppose there is a jar containing many gumballs, each with a unique number on it. The numbers range from 0 to 32 and there is an equal number of gumballs with each number. A student set out running an experiment with the following procedure: Pick five gumballs from the jar, calculate the mean of the numbers on the gumballs, write down the result on a piece of paper, and put the gumballs back to the jar. Repeat the process 499 times so altogether there are 500 means recorded.

So how is it compared to an approach with a sample size of 1, and without replacement, so he picks 2500 gumballs at once? Is not it a better estimation of the mean?

Is picking an N/m sample of sample size m, is better than picking an N sample of size 1 when estimating a population mean? In which case the variance will be higher?

• You say "each with a unique number on it" but then later imply that many balls could have the same number. I think you are misusing the word "unique". – alex.jordan Jan 19 '14 at 6:10
• I think an example like this is just supposed to be an example of how $\bar{x}$ can vary for fixed $n$ (5). There is no imperative to estimate the population mean. If there were, we can just calculate it ($16$), since your conditions say that each number appears the same number of times. – alex.jordan Jan 19 '14 at 6:24
• I agree. By the way the question was copied-pasted from here onlinestatbook.com/2/sampling_distributions/SampDist_v1.html , but I think implicit assumption that we dont know the real population, that they are uniformly distributed. So we cant say 16. – Emmet B Jan 19 '14 at 6:44
• If "the numbers range from $0$ to $32$" and "there is an equal number of gumballs with each number" then we definitely do know that the population average is $16$. – alex.jordan Jan 19 '14 at 8:01
• If you've accepted an answer, presumably you think it is good enough to upvote it. – alex.jordan Jan 19 '14 at 8:05

Let's first clearly define some terminology here.

• A draw consists of the process of taking a single gumball from the jar and observing its value.
• A sample consists of the process of taking five draws from the jar and observing each of their values.
• A draw with replacement consists of a draw for which the gumball is returned to the jar after observing its value.
• A draw without replacement consists of a draw for which the gumball is NOT returned to the jar.
• A sample with replacement consists of a sample for which the sample of five balls is returned to the jar.
• A sample without replacement consists of a sample for which no balls are returned to the jar.
• A sample mean is the arithmetic mean of the values observed from a single sample.

As the scenario is described, the draws are without replacement but the samples are with replacement. However, because we are told that there are "many" balls, and the proportion of balls that are labeled with a given number (from 0 to 32) is equal, we can assume that the sampling distribution of individual draws without replacement is approximately the same as the sampling distribution of draws with replacement; that is to say, because there are many gumballs, individual draws are assumed to be independent and identically distributed.

Now, under the assumption that samples are taken with replacement (as described in the given scenario), each sample is also independent and identically distributed. So the sampling distribution of the sample mean takes on a simple form that does not depend on the number of gumballs in the jar, only the number of samples taken (500) and the number of draws in each sample (5).

If we instead take samples without replacement, as you propose, then the samples are no longer independent and identically distributed, because the removal of previous samples means that subsequent samples are not taken from the same population. This makes the computation of the sampling distribution dependent on the total number of balls and their type.

Now, your question is whether sampling with replacement or without replacement gives a better estimate of the true population mean of the value on all the balls. The truth is, it depends on how many balls are in the jar and how many samples you take. If you take as many samples without replacement as are possible for the number of balls in the jar (e.g., the jar has 500 balls and you take 100 samples of 5 draws each), certainly that resultant sample mean will be best because you've observed all of the balls. But now suppose there are infinitely many balls in the jar--then it makes absolutely no difference--both will estimate the true population mean equally well, so long as you take the same number of samples in either scenario.

As you might expect, a single sample consisting of 2500 draws without replacement, as opposed to 500 samples with replacement of 5 draws without replacement, will be better at estimating the true population mean if the number of balls in the jar is at least 2500, but finite. But the sampling distribution is much more complicated to express in the non-replacement scenario.

• Good . how about both cases without replacement? ANy difference, 500 draws of 5 balls vs 2500 draws of 1 ball? So that in both cases you draw 2500 balls. – Emmet B Jan 19 '14 at 6:48
• Taking 500 samples of 5 draws each, versus 1 sample of 2500 draws, is the same if none of the balls is replaced at any time. The mean of the 500 sample means is the same as the sample mean of 2500 draws if all draws are without replacement. – heropup Jan 19 '14 at 6:53

As you suspect, the variance will be higher when you allow replacement of the balls. Replacement makes samples like $\{0,0,0,0,0\}$ more possible than they would be without replacement, since you might reuse the same balls. And samples like this have averages that are more extreme.

• how about both cases without replacement? 5 balls with 500 picks ( a total of 2500) vs 2500 single ball picks? Is it possible to say anything is better than the other? I think at least we can say 2500 single ball's distribution is more close to real distribution , but what about the mean and the error of the mean? – Emmet B Jan 19 '14 at 6:51
• @EmmetB After you pick the first 5 balls, are you then replacing them before picking the next? That is the scenario I was talking about. If you really never replace, then there is no difference. If you average the 500 averages from size-5 samples, you get the same result if you had just averaged all 2500 numbers in one go. – alex.jordan Jan 19 '14 at 7:59