# How many rolls do I need to determine if my dice are fair?

Roughly how many times do I need to roll a 6-sided die to feel confident that it's giving "fair" results? What about a 10-sided or 20-sided die?

Note that I will be actually manually rolling physical dice, this isn't just a textbook exercise. I'd like to minimize how long it takes me to perform this experiment with each die :)

I know this depends on my expected "confidence level" (95%? 99%?) If I choose a 95% confidence, for example, does that imply that 1 out of 20 fair dice will fail this test? Or that a single fair dice would fail the test 1 out of 20 times? If so, that sounds fairly high.

Are there standard techniques for doing this kind of a test?

Edit: It is beyond the scope of the math-focused question I've asked here, but I've explained more about the overall testing scenario over on the stats site here: http://stats.stackexchange.com/questions/14301/designing-a-test-for-a-psychic-who-says-he-can-influence-dice-rolls/14302#14302

-
Statistical tests checking if data came from a particular distribution should be used, e.g. Andreson-Darling test, Kolmogorov-Smirnov test. If you are Mathematica user, DistributionFitTest automates the testing. –  Sasha Aug 15 '11 at 17:17
You should also consider asking this question on stats.SE. –  Sasha Aug 15 '11 at 17:34
I'm surprised to see someone mentioning Kolmogorov-Smirnov before mentioning the simplest sort of chi-square test. –  Michael Hardy Aug 15 '11 at 17:36
Do you suspect a manufacturing/design defect? –  uncle brad Aug 15 '11 at 17:48
A decidedly low-tech approach (or perhaps high-tech, depending on your point of view) would be to write a simple simulation algorithm in your favorite programming language. Of course, the quality of the results would depend on the quality of the randomization algorithm you used, but there are many good ones available... –  ItsNotObvious Aug 15 '11 at 19:50

A chi-square test is the first thing that comes to mind: $$\sum\frac{(\text{observed} - \text{expected})^2}{\text{expected}}$$ If you roll the die $n$ times, the "expected" number of times you would see any particular outcome is $n/6$. If $n$ is large, this has approximately a chi-square distribution with 5 degrees of freedom. You reject the null hypothesis of fairness if the test statistic given above is large.

95% confidence does mean one out of twenty fair dice will fail.

See also this amazing analysis by a physicist of perhaps the most extensive experiment of this kind ever done: http://bayes.wustl.edu/etj/articles/entropy.concentration.pdf

-
I'm going to need a bit more hand-holding. How many rolls is "large"? 100 dice rolls? 10,000? How can I actually calculate the confidence level? Would it simplify things if I was only paying attention to the number of times a specific face came up, instead of trying to calculate based on all faces? –  BradC Aug 15 '11 at 19:30
"Large" in this context sometimes means you can make the approximation as close as you want by making the number large enough. How large is large enough depends on how close you want the approximation to be, and that leads into the practical problem. Two questions: How large must the same size be in order for the chi-square approximation to be practical?; and: How large must be the sample size be in order to be likely to detect "unfairness"? For the first question, there are rough rules of thumb, one of which is that each of the six outcomes should appear at least five times. –  Michael Hardy Aug 16 '11 at 0:57
Here's a paper that goes into more detail than I can right now: jstor.org/stable/2683047 –  Michael Hardy Aug 16 '11 at 0:58

You say this is for a test of paranormal abilities. So you have to ask your psychic what they think they can achieve. They might say one of the following:

I can throw a six more often than chance.
I can throw 1, 2, or 3 more often than chance.
I can throw a larger than expected total.

Whatever they say, get it in writing. This is a psychic you're dealing with.

Now you have to decide on your confidence level (I think 99% is reasonable here), and let your psychic choose the length of the test. Otherwise they might claim that they got tired (if there were a lot of tests), or that they didn't get into their stride (if there weren't).

Let's assume they claim to be able to throw sixes. If the die is fair, then the number of sixes in $n$ throws follows a binomial distribution, with mean $\mu = n/6$ and variance $\sigma^2 = 5n/36$. For large enough $n$ (which should certainly be the case here), the binomial distribution approximates the normal distribution, which for a one-tailed test at the 99% confidence level gives a cutoff of about $\mu + 2.326$ $\sigma$, or $n/6 + 0.867 \sqrt n$.

So now you can offer (say) the following choices:

$n = 100: 100/6 + 0.867*10 = 25$ sixes
$n = 400: 400/6 + 0.867*20 = 84$ sixes
$n = 900: 900/6 + 0.867*30 = 176$ sixes

Whatever the psychic decides, get it in writing. This is a psychic you're dealing with.

The psychic will (with probability 99%) fail the test, and will (with probability 100%) come up with something like "Yeah, but look at all those fours!" or "I never could figure out why Wednesdays don't work for me -- how about we do it again tomorrow?"

Let us know how it goes.

-

The statistical tests that check whether data came from particular distribution construct a function of sample points $S(x_1, \ldots x_n)$, called statistics. Assuming that data indeed came from the purported distribution, this statistics, itself being a random variable, follows certain distribution $\mathcal{D}_S$. The test consists in computing the value of statistics on your data sample, and verifying that the result is not too improbable. What constitutes "not too improbable" is controlled by the confidence level.

Notice that even when the samples are indeed from the purported distribution, the statistics with small probability can legitimately fall into the tail, resulting in a false negative.

When sample data are not from the purported distribution, then the distribution of statistics is different, and the outcome of the test becomes even less definitive.

So when the dice is indeed fair, and you would consider repeated your test many times, $95 \%$ confidence level means that about $5\%$ you would get false negatives.

-