# What percentage off a theoretical average is good enough to expect bias?

Say I roll a dice 600 times.

Theoretically, you should expect 100 sixes.

But, say, I only got 80. Would this be enough to expect bias?

I'm looking for a generally accepted percentage off, or a formula to calculate when you would expect it to be biased, but I'll happily receive anything else.

-
Instead of bias-ness, how about... bias? –  M Turgeon Feb 23 '12 at 16:45
Is that right? Ok. –  ACarter Feb 23 '12 at 16:50
You have to be careful: if you are observing a very long series of tosses, and see an anomalously low number of $6$'s in the last $100$, you can't accuse the thrower of having switched to loaded dice. It is statistically wrong, and also may get you beat up. Drug companies play this game. They fund a large number of studies, and only announce the results of the good ones, meaning good for them. –  André Nicolas Feb 23 '12 at 17:03

A simple chi-square test is often used for this.

The sum $$\sum \frac{(\text{observed} - \text{expected})^2}{\text{expected}}$$ means this: the "expected" number of times you see a "$1$" is $1/6$ of the number of times you throw the die; the "observed" number is how many times you actually get a $1$. See this article. There would be six terms in this sum.

If the die is unbiased, then this sum has approximately a chi-square distribution with $6-1=5$ degrees of freedom when the number of trials is large.

If this is so large that a chi-square random variable with $5$ degree of freedom would rarely be that large, then you reject the null hypothesis that the die is unbiased. How rare is "rare" is essentially a subjective economic decision. It's how frequently you get "false positives", i.e. how frequently you'd reject the null hypothesis when the die is actually unbiased.

There's a dumb stereotypical value of $5\%$ that gets used in medical journals. I.e. one false positive out of $20$ is OK; anything more is not. Using $1\%$ might be more sensible.

-
Thanks. So, for my example, I have $\frac{(80-100)\sqrt}{100}$. This gives 4. So anything more than 4% off 100 is bias? (Feel free to sort out my code if you want.) –  ACarter Feb 23 '12 at 17:02
@MichaelHardy I imagine that the reason for using a 5% threshold for significance is that a 5% threshold requires 400 people in your sample, whereas a 1% threshold needs 10,000 people. –  Chris Taylor Feb 23 '12 at 17:28
@MDCCXXIX : I don't see that you're making any sense. We're talking about throwing a die 600 times. There are no "people" in the sample, and you certainly don't need anything like 10000 trials to get a chi-square statistic that is in the rejection region at the 1% level. 100 is plenty for that. –  Michael Hardy Feb 23 '12 at 17:37
@ACarter : You haven't said how many times you got a 1, a 2, etc. So we've only got 6s and non-6s. The "expected" number of 6s is 100; the expected number of non-6s is 500. The "observed" numbers are 80 and 520. So the chi-square statistic would have only $1$ degree of freedom, and its value is $(80-100)^2/100 + (520-500)^2/500=4.8$. A chi-square random variable with $1$ degree of freedom has probability about $0.02846$ of exceeding $4.8$. So you'd reject the null hypothesis if you're willing to tolerate a $3\%$ chance of a false positive. –  Michael Hardy Feb 23 '12 at 17:43
@MichaelHardy Absolutely, it makes no difference if you're rolling a dice (although I don't fancy rolling a dice 600 times, let alone any larger number). It makes a huge difference if you're conducting an expensive medical study. If you want a higher confidence level, it costs more to run the study - so I dispute your assertion that the value of 5% is 'dumb'. (The figure of 10,000 is the sample size you would need to be confident that a binomial proportion is within 1% of its true value). –  Chris Taylor Feb 23 '12 at 17:53