When is random selection skewed (untrustworthy)?

Question

Imagine there is a population of 100 people, out of which 3 are to be randomly selected each day for alcohol testing.

After a month of such selections (after 20 selections), how many times somebody needs to be selected before I need to worry about the randomness in the process?

Real data: I got two people selected 3 times each; seven people selected 2 times, and a bunch of them selected 1 time (and another bunch never selected) in the previous month.

If the same person gets selected everyday for 20 days, there certainly (about 99.9999999999%) is something wrong.

What's the chance that there's something when a person is selected 5 times? What is he's selected 4 times? 3 times? What if two people get selected 4 times each? ...

When should I start thinking about making weighted selections?

How do I go about making these kind of calculations?

Answer 1

Given twenty testing dates in a month, 100 person population, and three people chosen without replacement for each testing date:

From an individual person's perspective, there is a $1-\frac{\binom{99}{3}}{\binom{100}{3}}= 0.03$ chance of being selected. The chance that he is selected exactly $k$ times out of $20$ trials would be $\binom{20}{k}0.03^k0.97^{20-k}$

This yields the following table for chances

$\begin{array}{|c|c|}\hline\text{picked exactly}~k~\text{times}&\text{probability}\\ \hline 0& \approx 0.5437\\ \hline 1& \approx 0.3364\\ \hline 2& \approx 0.0988\\ \hline 3& \approx 0.0183\\ \hline 4& \approx 0.0024\\ \hline 5& \approx 0.00024\\ \hline 6& \approx 0.000018\\ \hline \vdots\\ \hline\end{array}$

Treating each person as as a bernoulli variable then, we see that we should expect in $100$ people that over a month there should be around $54$ people never tested, around $33$ people tested once, around $9$ people tested twice, around $2$ people tested three times, and maybe a person or two tested more times than that.

Of course, that does not stop someone from having been tested six or more times, just that it is an unlikely scenario.

For each of these, the standard deviation is calculated as $\sqrt{np(1-p)}$, and we should expect most of the time to lie within two standard deviations of the mean.

$\begin{array}{|c|c|}\hline\text{picked exactly}~k~\text{times}&\text{expected range of number of people}\\ \hline 0& 49 - 59\\ \hline 1& 29-38\\ \hline 2& 7-13\\ \hline 3& 0-3\\ \hline 4& 0-1\\ \hline 5& 0-1\\ \hline 6& 0-1\\ \hline \vdots\\ \hline\end{array}$

To answer your actual question, if the data falls outside of the above ranges by a considerable margin, then perhaps start to worry, but keep in mind that being within two standard deviations is expected to occur only around $95\%$ of the time. There will be times that it falls outside of that range.