I didn't exactly know how to phrase the title of this question so a little more information..

I was conducting a small experiment with a class of secondary-school students to demonstrate the law of large numbers. Students were recording the result of independent dice rolls.

The theoretical probability for rolling any value on a dice (1 to 6) is of course ${1\over6}$ or ~16.67%.

I wanted to demonstrate that a small number of trials will yield results that may differ substantially from the theoretical probability but as we increase the number of trials the observed probability will converge towards the expected probability. Which I showed by collating all the students trials together.

The following is a table with the collated results from independent dice rolls.

data results

with 1146 trials the observed probability is within about 10% of the theoretical probability (The most being P(1) which is 9.42% away.

Is there a formula that determines how many trials must be done in order for the observed probability to be within a certain percentage of the expected probability?

For example, how many trials are needed to be confident the observed probability will be within 1% of the expected probability?

Is there a field of study for this? If so what is it called?

The reason I want to know is because I want to talk about randomness and bias with these students. Say a dice used favours a specific number but only slightly which in an experiment with a small amount of trials may not be evident.

  • 1
    $\begingroup$ Have you heard about hypothesis testing in statistics? If you did, this link may have what you are looking for stat.yale.edu/Courses/1997-98/101/chigf.htm $\endgroup$ – blabla zazoo Nov 8 '18 at 18:45
  • $\begingroup$ Your probability can always be from 0% to 100%, of course for a large number of trials, you are not likely to see such extreme values if the die is truly fair. The actual proportion for each face in your experiment will be pretty close to 1/6 for each face. Maybe the law of large number and central limit theorem are relevant and helpful here? $\endgroup$ – jdods Nov 8 '18 at 22:44

Let's just look at the proportion of die rolls that show the six face.

Let's define a random variable $X=1$ if we roll a six (we'll call this a success), and $X=0$ if we roll anything other than six (we'll call that a failure). Assuming that the die is perfectly fair, we get $P(X=1)=\frac16$ and $P(X=0)=\frac56$.

If we are to roll the die $n$ times, then each roll has an $X$ associated with it. We will call them $X_1,X_2,\ldots, X_n$. Note that this is just a list of $1$'s and $0$'s, and the exact values of which are determined by the random sequence of die rolls. (The technical term is that we call these independent and identically distributed Bernoulli trials.)

Now to know the total number of sixes we get in our sequence of $n$ rolls, we just add up all the $X_i$'s, $Y=\sum_{i=1}^nX_i$.

$Y$ is known to be a "binomial random variable" with parameters $n$ (number of trials) and $p=\frac16$ (probability of success).

Now to know the proportion of our trials that are sixes, we calculate $\frac Yn$. For example if out of 105 die rolls, you get 19 sixes, then $Y=19$ and $\frac Yn=\frac{19}{105}\approx 0.181=18.1\%$.

We'll call $\frac Yn=\hat{p}$ to shorten the notation and to call attention to the fact that it is a rough estimate of the true $p=\frac16$.

The law of large numbers says that $\hat p$ converges to $p$ as $n$ goes to infinity. This is a special type of convergence known as "convergence in probability". It just means that the actual value you get for $\hat p$, even with a very large number of die rolls, is random and you could hypothetically get zero sizes, or even all of the rolls could be sixes. The likelihood of that happening is very low though. The greater the number of trials, the lower the likelihood of your calculated $\hat p$ being too far from the true $p$ value.

The central limit theorem says that $\hat p$ is approximately normally distributed with mean $p$ and variance $\frac {p(1-p)}{n}$. This is often notated as $\hat p\sim N\left(p,\frac {p(1-p)}{n}\right)$. Notice that as $n$ gets large, the variance $\frac {p(1-p)}{n}$ gets smaller, approaching zero. You can think of the variance as a measure of how likely your calculated $\hat p$ is to be far away from $p$.

Now, since all this is random, you cannot actually get any guarantee of a specific number of die rolls always keeping your calculate proportion of sixes within, say, 1% of the true expected probability of six. However, you can set a probability, say 5%, where we want to say: "There is at least a 95% chance that my calculated proportion of sixes will be within 0.001 of the true probability of six." In probability notation this is $P(p-0.001<\hat p <p+0.001)\geq 95\%$.

Recalling that $\hat p$ is approximately normally distributed with mean $p$ and variance $\frac {p(1-p)}{n}$, we can actually use this to calculate a number of trials $n$ to get the bound that we want.

We won't worry about the "$\geq$" in the equation above. Let's just replace it by an equals sign. Just know that increasing the number of trials will always increase the probability of being within our desire bounds of the actual true $p$ value.

This gives $P(\frac16-0.001<\hat p <\frac16+0.001)= 0.95$. Now, we also know that the normal distribution is symmetric, so this mean that we are creating two 2.5% tails in the above probability equation. This means that $P(\hat p \leq \frac16-0.001)=0.025$. Using the inverse of the cumulative normal distribution, we can calculate this. We need to find the 2.5$^{th}$ percentile for this normal distribution and set it equal to $\frac16-0.001$. The $q^{th}$ percentile of the normal distribution with mean $\mu$ and standard deviation $\sigma$ (variance $\sigma^2$) is given in R by $\texttt{qnorm($q$,mean=$\mu$,sd=$\sigma$)}$. You can also use normal probability tables, but I teach my students to use R.

Furthermore it turns out that we can use the standard normal instead (which is mean 0 and variance 1) and have $\texttt{qnorm($q$,mean=$\mu$,sd=$\sigma$)}=\mu+\texttt{qnorm($q$,mean=$0$,sd=$1$)}*\sigma$

So for our parameters, $q=0.025$, $\mu=p=\frac16$ and $\sigma=\sqrt{\frac {p(1-p)}{n}}$. So we get $$\frac16+(-1.96)\frac{\sqrt{5}}{6\sqrt n}.$$ Now we set this equal to the lower bound (left side) of our probability interval $$\frac16+(-1.96)\frac{\sqrt{5}}{6\sqrt n}=\frac16-0.001$$ and solve for $n$ to get $n=533,555.6$. Thus you need more than five hundred thousand die rolls to get within 0.001 of the true probability 95% of the time.

In general, let $\epsilon$ be your error tolerance. Solving for $n$ gives $$n\geq \frac1{\epsilon^2}\texttt{qnorm(1-$\alpha$/2)}^2p(1-p).$$

If you want to be 99% certain that your calculated probability is within 0.0001 of the true value $\frac16$ (note that $0.0001/0.166667 \approx 0.06\%$), then you'll need at least 92,151,342 die rolls!

How to get within 1%?
To answer your question, to be within 1% of $p$ 95% of the time, you'll need to perform about 192,073 die rolls. To raise your confidence to 99%, you'll need to do 331,745 die rolls!

Note that even though, say 5% of the time your calculated $\hat p$ will be further from the true $p$ than you want it to be, it is still not very likely to be too much further away.

I just ran a few simulations in R. For 1,146 die rolls, the 95% interval is 0.145 to 0.188, so your data is pretty typical. This 95% interval is a $\pm13\%$ deviation from 1/6, but the mean deviation is about 7%. If you go up to 5,000 die rolls, the 95% interval is 0.156 to 0.177 which represents a $\pm6\%$ deviation from 1/6, but the mean deviation is about 3%. If you go up to 10,000 die rolls, the 95% interval is 0.159 to 0.174 which represents a $\pm4\%$ deviation from 1/6, but the mean deviation is about 2%.

  • 1
    $\begingroup$ Thank you for taking the time to write this! It means a lot to me $\endgroup$ – Danoram Nov 14 '18 at 20:29
  • 1
    $\begingroup$ Glad to be of service. I just happened to be lecturing on this in class at the moment. My first bounty! $\endgroup$ – jdods Nov 14 '18 at 22:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.