I have been presented with a study which compares failure rates in two samples. The fail rates are 11% and 17%, but the sample sizes are different: 90 and 270 respectively.

I have an intuition that the difference in sample size is potentially significant and it reduces my confidence that I can infer anything at all from the comparison.

I am wondering if there is some mathematical way to calculate the following:

  • How large a sample size should be
  • Whether the difference in sample sizes between two batches is significant

Can anyone point to some answers or reading sources for these kinds of problems?

  • $\begingroup$ Well...let's say (from the second example) that we imagine the probability of failure to be $p=.17$. Then the probability of getting $.11\times 90=10$ (I'm guessing) or fewer fails in the smaller sample is about $8.4\%$ which is low but hardly shocking. $\endgroup$ – lulu Nov 28 '17 at 21:55

The sample sizes are known to be different, so there is no direct issue of statistical significance as to difference in sample sizes. However, it does make sense to wonder whether the sample sizes you mention are too small reliably to detect a difference between the two failure rates you mention.

If your are testing that the two failure rates are the same $H_0: \theta_1 = \theta_2$ against the alternative hypothesis $H_a: \theta_1 \ne \theta_2$ or $H_a: \theta_1 < \theta_2,$ then may want to know what sample size $n$ is necessary to detect a difference as great as $\delta = \theta_2 - \theta_1 = .17 - .11 = .06$ with power $.90.$ Power is the probability of rejecting $H_0$ when $\delta = .06.$ The answer depends on the significance level $\alpha$ (often 5%) you use for the test. It would also be different, depending on whether you use a one- or two-sided alternative.

The following output from Minitab statistical software indicates that, for a two-sided 5% level test, having power 90% would require $n_1 = n_2 \approx 700.$ So you are correct to suspect that sample sizes $n_1 = 90$ and $n_2 = 270$ are not large enough for most practical purposes.

   Test for Two Proportions

Testing comparison p = baseline p (versus ≠)
Calculating power for baseline p = 0.11
α = 0.05

              Sample  Target
Comparison p    Size   Power  Actual Power
        0.17     701     0.9      0.900102

The sample size is for each group.

In the figure below, the dotted line is for 90% power $(n = 701)$ and the solid line for 80% power $(n=524).$

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.