Calculating a sample size based on a confidence level

Question

It's been a while since my last statistics class...

I have $404$ files that went through some automated generation process. I would like to manually verify some of them to make sure that their data is indeed correct. I want to use probability to help me out so that I don't need to check every single file.

How would I calculate what sample size I should use to reach a certain confidence level?

For example, if I would like to say with $95\%$ confidence that the files are correct, how many of them do I have to check?

I found an online calculator, but I'm not entirely sure what I should put for the confidence interval. Say I put 20% and leave the confidence factor at $95\%$. I get a sample size of 23. Let's say now that I tested 23 random files and all of them were fine. Does that mean that "I can be $95\%$ confident that 80% to 100% of the files are correct"?

Does this mean, then, that for my original question, I would need to use a 99% confidence level with a 4% confidence interval, then I would need to verify that the 291 files (the sample size it gave me) are all correct. And only then I can say with 95% confidence that the files are correct? (99% +- 4% = 95% to 100%)

It also mentions something about percentages which I'm not quite clear on... does the fact that most (i.e. 100%) of the files I test are valid (since if I found an invalid one, I would stop the whole process and examine my generation process for errors) mean that I can use a smaller sample to get the same confidence factor? If so, how would I calculate it?

Answer 1

It's not surprising you're a bit confused; understanding what's really going on with confidence intervals can be tricky.

The short version: If you don't want to check all the files you have to choose two different percentages: the confidence level (95% in your example), and how far off you're willing to be at that level (20% in your example). These percentages refer to two different quantities, and so it doesn't make sense to add or subtract them from each other. Once you've made these choices then I think it is fine to use the online calculator to get a sample size.

If you want more detail on what's going on, here's the explanation: You're trying to estimate the true percentage of files that have correct data. Let's call that percentage $p$. Since you don't want to calculate $p$ exactly, you have to choose how far off you are willing to be with your estimate, say 20%. Unfortunately, you can't even be certain that your estimate of $p$ will be within 20%, so you have to choose a level of confidence that that estimate will be within 20% of $p$. You have chosen 95%. Then the online calculator gives you the sample size of 23 you need to estimate $p$ to within 20% at 95% confidence.

But what does that 95% really mean? Basically, it means that if you were to choose lots and lots of samples of size 23 and calculate a confidence interval from each one, 95% of the resulting confidence intervals would actually contain the unknown value of $p$. The other 5% would give an interval of some kind that does not include $p$. (Some would be too large, others would be too small.) Another way to look at it is that choosing a 95% confidence interval means that you're choosing a method that gives correct results (i.e., produces a confidence interval that actually contains the value of $p$) 95% of the time.

To answer your specific questions:

"Does that mean that 'I can be 95% confident that 80% to 100% of the files are correct'?" Not precisely. It really means that you can be 95% confident that the true percentage of correct files is between 80% and 100%. That's a subtle distinction.

"And only then I can say with 95% confidence that the files are correct? (99% +- 4% = 95% to 100%)?" No, this is confusing the two kinds of percentages. The 99% refers to 99% of all confidence intervals constructed if you were to construct lots of them. The 4% refers to an error margin of $\pm$ 4% for the files.

One other thing to remember is that the sample size estimator assumes that the population you're drawing from is much, much larger than the size of the sample you end up going with. Since your population is fairly small you can get away with a smaller-sized sample with the same level of confidence. The determination of exactly how small, though, is a much more difficult calculation. It's beyond what you would have seen in a basic statistics class. I'm not sure how to do it; maybe someone else on the site does. (EDIT: Even better: take Jyotirmoy Bhattacharya's suggestion and ask on Stats Stack Exchange.) But this is the only justification for being able to use a smaller sample size than 23 - not the fact that you would abort the confidence interval calculation if you found anything other than 100% for your sample's estimate of the true value of $p$.

Answer 2

Let's forget about confidence intervals for a while and try to think of your sample size problem from first principles. This might also answer @Mike's point about finite samples.

Suppose among your $N$ files $M$ are good and $N-M$ are bad. If you pick a random sample of $n$ files the probability that all of them are good is:

$$p(n,M) = {{M \choose n} \over {N \choose n}}$$

[This is a special case of the hypergeometric distribution. You can calculate it in Excel as HYPGEOMDIST(n,n,M,N) or in R as dhyper(n,M,N-M,n)]

$p(n,M)$ will be smaller for larger $n$. A large sample is more likely to uncover bad files even if they are rare.

$p(n,M)$ will be larger for larger $M$. Rarer bad files are, the less likely are they to be caught by a sample of a given size.

How large an $n$ should you choose?

If you knew $M$ then you could plot $p(n,M)$ and choose a $n$ large enough to put $p(n,M)$ below the threshold of error you are willing to tolerate.

But the whole point of the exercise is that you don't know $M$. Now it becomes a question of how optimistic or pessimistic you are. If you are an optimist then you will think that your process will either succeed or fail very badly. So if $M$ is not $N$ then it is a very small number. In this case choosing a small $n$ will let you reach your error tolerance.

If you are a pessimist you will think that if your process fails it will produce a bad file only occasionally, i.e. $M$ will be large but not equal to $N$. In this case you will be forced to take a large sample to reach any given error tolerance.

If you want to be formal you could try to codify you pre-sampling beliefs about the value of $M$ in terms of a probability distribution. [I have not seen anyone really do it, though I have seen people sometimes pretend to do so.] Let the probability mass function be $f(m)$. Then the probability of a sample of size $n$ having all good files even if some files are bad is:

$$\pi(n)={1 \over {1-f(N)}}\sum_{m=0}^{N-1} p(n,m)f(m)$$

Now you can choose a large enough $n$ to keep $\pi(n)$ below your error tolerance.

Some additional points:

• I assumed that the 404 files are all you care about. On the other hand if you care about the process that generated the files, then you should model the process as generating bad files with a probability $q$ and instead of $p(n,M)$ have $p(n,q)=(1-q)^n$.

• I assumed from your question that you are interested only in the two possibilities $M=N$ and $M \neq N$. The extension of the approach above when you care about how many files are bad is left as an exercise.

• The "error tolerance" mentioned about should not be a number picked from the air or chosen by convention. It should be the result of considering the costs of inspections, the damages that would result from missing a bad file, and your attitude towards risk.

Finally, what about confidence intervals? They are a barbaric relic of the past. In most situations they don't answer any interesting questions.