# Probability - analyzing "randomness" of data

Forgive me, I am a probability novice and am looking for a little guidance. My question is based on real-world data. I have obscured it a bit for confidentiality reasons but the spirit of the question is the same. Okay, here is the setup: Suppose that 100 students from 37 different schools have applied to take part in a math camp. We are told that 188 students will be chosen randomly from the 3700 total applications. Suppose the following number of students are selected from each school:

1. 7
2. 8
3. 1
4. 5
5. 11
6. 3
7. 6
8. 15
9. 3
10. 7
11. 43
12. 1
13. 1
14. 2
15. 1
16. 23
17. 4
18. 3
19. 5
20. 5
21. 6
22. 2
23. 16
24. 1
25. 1
26. 2

(Schools 27-37 have 0 students selected)

Now, I am suspicious about the large number of students chosen from School 11 (43 students), so I wish to analyze this data to determine the likelihood that the applications were randomly selected. Mathematically I believe this equates to determining whether or not the data follows a normal distribution.

My attempt at the solution is the following. Since there are 37 different schools, and each school submitted the same number of applications, I would expect 188/37 ~ 5 students chosen from each school (i.e. this is the mean of my random variable). I would like to determine a range such that - if the students were randomly selected - "there is a 99% probability that the number of students chosen from each school would be between x and y" (so that I can see whether 43 falls into this range). However I am unsure what to use for the standard deviation.

• You could attempt an analytic solution, but why not try a computer simulation first? Dec 15 '15 at 0:28
• Such a huge deviation assures you that the selection wasn't random and there was human factor involved. It IS possible that their random number generator got royally messed up though.
– A.S.
Dec 15 '15 at 2:48

If 188 students were truly selected at random from a pool of 3700, you will be able to calculate a $p$-value: that is, the probability under the null hypothesis (of random selection) that you see an outcome as "extreme" as the one you found. One possible definition of "extreme" is "at least one school had at least 43 students chosen". To calculate the probability of this latter event, you can apply a symmetry argument to bound the prob by $37 \times P(N_1\ge 43)$, where $N_1$ is the number of students selected from school 1.

Now $N_1$ is a random variable having a hypergeometric distribution. Using tail bound (10) in the paper http://arxiv.org/pdf/1311.5939v1.pdf , the probability $P(N_1\ge 43)$ can be bounded as follows: $$P(N_1\ge 43)= \sum_{i=k}^n {{M\choose i}{N-M\choose n-i}\over {N\choose n}} \le \exp(-2t^2n),$$ with $N=3700$, $M=100$, $n=188$, $k=43$, and $t$ satisfying $E(N_1) + tn = k$. I get $E(N_1)=188/37$ and $t=.201696$, which leads to the upper bound on the $p$-value: $$p \le 37\times P(N_1\ge 43)< .00000023.$$ (please check my work.) Given how tiny this $p$-value is, I think it is safe to reject the null hypothesis of random selection.

• That bound look suspiciously like the regular bound Chernoff bound for the binomial (which is more dispersed than hypergeometric). Is that the case or is there an improvement?
– A.S.
Dec 15 '15 at 7:53
• @A.S. Yep, it's the Chernoff-Hoeffding inequality. The bound for the binomial case is found in Hoeffding's 1963 paper "Probability Inequalities for Sums of Bounded Random Variables"; he proves later in that paper that the same bound holds for the hypergeometric case (viewed as the sum of a random sample without replacement from a finite population). Dec 15 '15 at 18:31
• Then I am completely unclear as to why the note you linked has "ending insanity" as a sub-title, since the bound trivially follows from the binomial case. I would have expected some tighter bounds give such a dramatic title.
– A.S.
Dec 15 '15 at 18:34
• @A.S. Agreed, there's no insanity to be ended except the author's frustration in not finding a reference that collects results about the hypergeometric. He does admit that the paper has no original math in it. Dec 15 '15 at 18:48
• @grand_chat Thank you for pointing me in the direction of the hypergeometric distribution, it was very helpful. Dec 15 '15 at 19:53

I do not think you can take any sample of data and, simply because there is an outlier in the sample, draw any conclusion, beyond a human emotion that it's suspicious. Mathematically it's just data from a random process.

If you had some data to correlate the values with that might demonstrate some unlikely aspect of the outlier. For example, a previous round of values that showed how the previous and current data should distribute relative to each other.

But on it's own the outlier has no real significance.

Also note that by definition the process involved is a "choice". Students were chosen, which means that unless a systematic process was defined that can be audited, it was left up to humans to decide. The inevitability of wildly different results is almost guaranteed in this case.

From your description of the circumstances it would take little more than a very good maths teacher or teachers to justify the positive outliers and very bad maths teachers to justify the negative outliers.

I think any conclusion drawn from just considering the numbers of students per school would be totally unreliable.

The hypothesis you want to test is: All schools have an equal probability of having a student selected.

For a single school the number of students selected has a binomial distribution with $n=188$ and $p=1/37$. The expected value of students selected is $188/37$ and the variance of students selected is $188(1/37)(36/37)$.

Using the normal approximation to the binomial distribution there is a 99% probability that the number of students $X$ in a given school is between 4.7 and 5.5.

But this is just for one school. Since you have a lot of schools if you use this test you are bound to have a few schools which by chance got an abnormally large number of students selected. A school having more than 5.5 students chosen isn't automatically reason to suspect that the schools don't have equal chances.

This is a problem of testing multiple hypotheses at once. If you want to have a statistical significance of 99% for testing all the schools together then an individual school must meet a stronger criteria to be deemed suspicious.

If the probability of a false negative for one school is $q$ then when you test 37 schools the probability of a false negative occurring is $1-(1-q)^{37}$.

You want $1-(1-q)^{37} = 0.01$ which corresponds to $q=0.0003$.

In other words, if any school has a number of students which is extreme enough that its probability of occurring is 0.03% then this is sufficient evidence that you can reject the hypothesis with a significance of 0.01.

Since 0.03% is so small if you continue to use the normal approximation to the binomial distribution it will give you figures for a negative number of students. This indicates that the normal approximation isn't good enough so instead you can add up the probabilities of $X=188$, $X=187$, $X-186$, ect. until they add up to more than 0.0003 and that is the number of students selected required for you to consider the school "suspicious".

It is a typical multihypergeometric model, but close to the multinomial one, which you may try the classical chi-square goodness-of-fit test to test the null hypothesis $$H_0: p_1 = p_2 = \ldots = p_{37} (= \frac {1} {37})$$