# How many rolls do I need to determine if my dice are fair?

Roughly how many times do I need to roll a 6-sided die to feel confident that it's giving "fair" results? What about a 10-sided or 20-sided die?

Note that I will be actually manually rolling physical dice, this isn't just a textbook exercise. I'd like to minimize how long it takes me to perform this experiment with each die :)

I know this depends on my expected "confidence level" (95%? 99%?) If I choose a 95% confidence, for example, does that imply that 1 out of 20 fair dice will fail this test? Or that a single fair dice would fail the test 1 out of 20 times? If so, that sounds fairly high.

Are there standard techniques for doing this kind of a test?

Edit: It is beyond the scope of the math-focused question I've asked here, but I've explained more about the overall testing scenario over on the stats site here: https://stats.stackexchange.com/questions/14301/designing-a-test-for-a-psychic-who-says-he-can-influence-dice-rolls/14302#14302

• Statistical tests checking if data came from a particular distribution should be used, e.g. Andreson-Darling test, Kolmogorov-Smirnov test. If you are Mathematica user, DistributionFitTest automates the testing. – Sasha Aug 15 '11 at 17:17
• You should also consider asking this question on stats.SE. – Sasha Aug 15 '11 at 17:34
• I'm surprised to see someone mentioning Kolmogorov-Smirnov before mentioning the simplest sort of chi-square test. – Michael Hardy Aug 15 '11 at 17:36
• Do you suspect a manufacturing/design defect? – uncle brad Aug 15 '11 at 17:48
• The statistical test depends on the defect you are looking for. For example, two opposite faces more frequent could mean a slightly smaller dimension in that axis. Or less edge rounding on the face opposite a high frequency face. The statistical test will be more powerful if derived from a physical defect model. – richard1941 May 7 '16 at 3:50

A chi-square test is the first thing that comes to mind: $$\sum\frac{(\text{observed} - \text{expected})^2}{\text{expected}}$$ If you roll the die $n$ times, the "expected" number of times you would see any particular outcome is $n/6$. If $n$ is large, this has approximately a chi-square distribution with 5 degrees of freedom. You reject the null hypothesis of fairness if the test statistic given above is large.

95% confidence does mean one out of twenty fair dice will fail.

See also this amazing analysis by a physicist of perhaps the most extensive experiment of this kind ever done: http://bayes.wustl.edu/etj/articles/entropy.concentration.pdf

A further refinement of the chi square test would be to note that each outcome of a roll has an opposite face. If one outcome is unusually high, the opposite face should be unusually low. Thus, it is the difference between opposite face frequencies that detect unbalance in the die. You could create a simulation in a simple spreadsheet and find the confidence limits by Monte Carlo.

• I'm going to need a bit more hand-holding. How many rolls is "large"? 100 dice rolls? 10,000? How can I actually calculate the confidence level? Would it simplify things if I was only paying attention to the number of times a specific face came up, instead of trying to calculate based on all faces? – BradC Aug 15 '11 at 19:30
• "Large" in this context sometimes means you can make the approximation as close as you want by making the number large enough. How large is large enough depends on how close you want the approximation to be, and that leads into the practical problem. Two questions: How large must the same size be in order for the chi-square approximation to be practical?; and: How large must be the sample size be in order to be likely to detect "unfairness"? For the first question, there are rough rules of thumb, one of which is that each of the six outcomes should appear at least five times. – Michael Hardy Aug 16 '11 at 0:57
• Here's a paper that goes into more detail than I can right now: jstor.org/stable/2683047 – Michael Hardy Aug 16 '11 at 0:58

You say this is for a test of paranormal abilities. So you have to ask your psychic what they think they can achieve. They might say one of the following:

I can throw a six more often than chance.
I can throw 1, 2, or 3 more often than chance.
I can throw a larger than expected total.

Whatever they say, get it in writing. This is a psychic you're dealing with.

Now you have to decide on your confidence level (I think 99% is reasonable here), and let your psychic choose the length of the test. Otherwise they might claim that they got tired (if there were a lot of tests), or that they didn't get into their stride (if there weren't).

Let's assume they claim to be able to throw sixes. If the die is fair, then the number of sixes in $n$ throws follows a binomial distribution, with mean $\mu = n/6$ and variance $\sigma^2 = 5n/36$. For large enough $n$ (which should certainly be the case here), the binomial distribution approximates the normal distribution, which for a one-tailed test at the 99% confidence level gives a cutoff of about $\mu + 2.326$ $\sigma$, or $n/6 + 0.867 \sqrt n$.

So now you can offer (say) the following choices:

$n = 100: 100/6 + 0.867\times10 = 25$ sixes
$n = 400: 400/6 + 0.867\times20 = 84$ sixes
$n = 900: 900/6 + 0.867\times30 = 176$ sixes

Whatever the psychic decides, get it in writing. This is a psychic you're dealing with.

The psychic will (with probability $99\%$) fail the test, and will (with probability $100\%$) come up with something like "Yeah, but look at all those fours!" or "I never could figure out why Wednesdays don't work for me -- how about we do it again tomorrow?"

Let us know how it goes.

The chi-squared test mentioned above is the correct approach. However, in order to estimate the appropriate # of dice rolls for testing fairness you need to decide:

1) the desired power of your test (= probability of correctly detecting a biased die), and

2) the effect size you wish to detect with confidence.

A standard value for power is 80%. Typical effect sizes for a chi-squared test range from 0.1 (small) to 0.5 (large).

See this website for a quick tutorial on determining sample sizes for a chi-squared test using the free R language.

Choosing significance level = .05, power = 0.80, degrees of freedom = # sides - 1, and effect size = 0.5, I find the following sample sizes to test the fairness of 6-sided, 10-sided, and 20-sided dice:

6-sided: 52 rolls

10-sided: 63 rolls

20-sided: 83 rolls

The statistical tests that check whether data came from particular distribution construct a function of sample points $S(x_1, \ldots x_n)$, called statistics. Assuming that data indeed came from the purported distribution, this statistics, itself being a random variable, follows certain distribution $\mathcal{D}_S$. The test consists in computing the value of statistics on your data sample, and verifying that the result is not too improbable. What constitutes "not too improbable" is controlled by the confidence level.

Notice that even when the samples are indeed from the purported distribution, the statistics with small probability can legitimately fall into the tail, resulting in a false negative.

When sample data are not from the purported distribution, then the distribution of statistics is different, and the outcome of the test becomes even less definitive.

So when the dice is indeed fair, and you would consider repeated your test many times, $95 \%$ confidence level means that about $5\%$ you would get false negatives.