Sample size for estimating a probability

Assume I want to pass a test with 98 % probability, or 95 % probability or 90 % probability or even as low as with 75 % probability.

We say that I will take the test if the probability of success is at least 75 %.

The test has 80 questions. (It is the OCAJP 8 test.) I have a practice test with 77 questions. Now if I'm lazy I don't answer all 77 questions on the practice (mock) test but only as many as I need to estimate my probability of success.

For instance, I answer 4 mock questions and get 3 answers right, then it is too little sample size to assume that I will pass the test (where passing score is 65 %) and too little sample size to generalize to 75 %.

I know there is a "magic sample size" of about 30 questions that seems reasonable here. If I answer 30 questions and get 28 or 29 answers right then it will seem "reasonable" that I will get at least 65 % on the real test.

But how should I make a more exact and correct estimate? How many questions do I have to answer to generalize my mock score?

• This is an interesting question! However, I don't understand the meaning of the first two sentences. What grade do you need to pass the test? 50%? – Michael R Jan 14 '17 at 19:24
• Passing is 65 % – Niklas Jan 14 '17 at 19:26
• The answer depends on both the sample size of the practice exam and the proportion of correctly answered question on the practice exam. – Michael R Jan 14 '17 at 20:49

First, we need to make two important assumptions: 1) all questions on a given exam are equal in terms of difficulty, and 2) the questions on the practice and real exams are equal in terms of difficulty.

Let $X_i$ be a Bernoulli random variable that denotes whether or not the $i_{th}$ answer on the practice exam is correct, i.e. $X_i = 1$ if the $i_{th}$ answer is correct and $X_i = 0$ if the $i_{th}$ answer is wrong.

Let $p$ denote the proportion of correctly answered questions on the practice exam. Then $\hat{p}$, the MLE of p, is equal to $\hat{p} = \frac{\sum_{i=1}^{n}X_i}{n} = \bar{X}_{n}$.

Let $Y$ denote the total number of correct answers on the real exam. Then $Y \approx 80\hat{p}$. Since the minimum passing grade is 65%, we want to find:

$P(Y\geq52) = P(80\hat{p} \geq 52) = P\big(\hat{p}\geq\frac{52}{80}\big) = P\big(\hat{p}\geq0.65\big)$

By the CLT, $\frac{\hat{p}-E(\hat{p})}{\sqrt{Var(\hat{p})}} \overset{d}{\to} Z$. Furthermore, $E(\hat{p}) = \bar{X}_n$ and $Var(\hat{p}) = n\bar{X}_n(1-\bar{X}_n)$, which means that we get:

$P\bigg(Z \geq \frac{0.65-\hat{p}}{\sqrt{Var(\hat{p})}} \bigg) = P\bigg(Z \geq \frac{0.65-\bar{X}_n}{\sqrt{n\bar{X}_n(1-\bar{X}_n)}} \bigg)$

In order to pass with probability $(1-\alpha)100\%$, we need:

$P\bigg(Z \geq Z_{\alpha} \bigg) = 1-\alpha$, which means that we need: $Z_{\alpha} = \frac{0.65-\bar{X}_n}{\sqrt{n\bar{X}_n(1-\bar{X}_n)}} \rightarrow n = \Bigg(\frac{0.65-\bar{X}_n}{Z_{\alpha}\sqrt{\bar{X}_n(1-\bar{X}_n)}}\Bigg)^2$

Note: in this formula, $n$ is number of questions on the practice exam, while $\bar{X}_n$ is the proportion of correctly answered questions on the practice exam.