I've poked about in some other questions, and I'm no sure how to deal with my problem and my knowledge of statistics has atrophied. Particularly that I'm trying to choose a sample size for a population that I don't know the size of (potentially infinite, but it could be 10,000's or 100,000's or more). How do I choose a sample size that will give me a meaningful answer.

Is it reasonable just to plug in a very large number, and see what comes out - does it approach a limit?

My real world problem is this:

I have two computer systems (Able and Baker). My user community believes Able is faster than Baker. I can run a simple test on both, and see how long it takes to run one each. However, there are inconsistencies in performance (probably do to the network, which will have spikes in activity and I unfortunately can't removed from the test).

Baker will be running for years into the future, so I have no idea how many transactions will run in it over its lifetime.

Assuming the performance issues caused by the network are random, how many tests do I have to run each on Able and Baker to to be 90% confident that Able is faster than Baker?

Perhaps I'm asking the wrong question? Should I just be finding the average of a 100 tests on Able and 100 tests on Baker and compare? Can I make than number 100 smaller (to say like 20)

  • 1
    $\begingroup$ It is more or less impossible to answer your question without making statistical hypotheses. $\endgroup$ Commented Mar 23, 2011 at 20:39
  • $\begingroup$ You might also get good answers at stats.stackexchange.com $\endgroup$
    – user940
    Commented Mar 24, 2011 at 2:05
  • $\begingroup$ The Z-values for confidence levels are: 1.645 = 90 percent confidence level 1.96 = 95 percent confidence level 2.576 = 99 percent confidence level $\endgroup$
    – user106469
    Commented Nov 8, 2013 at 19:57

1 Answer 1


1.) A surprising result that we encounter in a first statistics course is that the quality of a typical estimate doesn't depend (much) on the population size, but mainly the sample size.

For instance, a sample average based on ten data points is equally accurate whether the population size is 1000, 1000000, or infinity.

You probably needn't worry about population size.

2.) Do you need 100 test runs for each program or will 20 suffice? This partly depends on your needs. I mean, is it important to you to know whether the two machines differ on average by one minute, one second, one millisecond? Where do small differences stop being important to you? You need to consider this question first because of the tradeoff between "quality of statistical test" and "required sample size".

The required sample size for your problem will likely look something like
$$n={2(1.645)^2 \sigma^2\over E^2}.$$

Let's plug in some made up numbers, just to get an idea what's going on. Suppose that the standard deviation of performance times is $\sigma = 1$ second. And suppose we want to be 90% sure to detect an average difference of 0.5 of a second. Then $n=2(1.645)^2/(1/2)^2=21.64\approx 22$. We should take a sample of about 22 runs from each machine.

On the other hand, if you wanted to detect an average difference of 0.25 of a second, the required sample size goes up by a factor of four, that is, $n=86.59\approx 87$. To double the accuracy, you need four times as many sample points.

Concrete information on both inherent variability and on required accuracy are part of the sample size calculation.

More details on statistical tests and sample size calculations can be found in most introductory statistics textbooks. Good luck!

  • $\begingroup$ I'm trying to use this formula. My problem is where does the 1.645 value come from? I guess it is linked to the 90 %, but how? I found at other places (stats.sec, maths.sec, wikipedia) a value named z (or zeta) alpha. Is it the same? How to calculate it? $\endgroup$ Commented Mar 21, 2013 at 9:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .