# Type-I vs. type-II error in statistical hypotheses testing

Let us consider standard statistical hypotheses testing:

$$\alpha=P\{\text{type}-I \text{ error}\}=P\{\text{Rejecting } H_0 \text{ when }H_0\text{ is true}\}$$

and

$$\beta=P\{\text{type}-II \text{ error}\}=P\{\text{Accepting } H_0 \text{ when }H_1\text{ is true}\}.$$

My question is as follows: could you give an example of $$\alpha$$ and $$\beta$$ when tossing 10 coins so that I can see that it does not hold this equation:

$$\alpha=1-\beta.$$

More specifically, compose an outcome so that inequality between $$\alpha$$ and $$1-\beta$$ is seen.

The issue is that whether $$\alpha = 1-\beta$$ in general depends on the construction of the hypothesis test--specifically, the test statistic and the rejection region. Certainly you could conceivably choose a statistic and rejection region such that this relationship could be satisfied, but that doesn't make it a useful test, nor does it make that relationship true in general. It is a bit like saying, $$x + y = 5$$ could be satisfied by $$x = 2$$, $$y = 3$$, but that doesn't mean $$x + y$$ always equals $$5$$.

In any case, suppose we take your ten coin example and make it more concrete. Instead of ten coins, say we have a single coin that we toss $$n = 10$$ times, and we are interested in the random number of heads $$X$$ obtained. Under the reasonable assumption that each trial is independent and identically distributed, we an impose a distributional assumption that $$X \sim \operatorname{Binomial}(n = 10, p = \theta),$$ where $$\theta$$ is some real number between $$0$$ and $$1$$ representing the probability of obtaining heads on any single toss. In this way, we can construct a hypothesis test regarding the fairness of the coin; i.e., $$H_0 : \theta = 0.5, \quad \text{vs.} \quad H_a : \theta \ne 0.5.$$ Intuitively, if $$X$$ is very small or very large (e.g., $$X = 0$$ or $$X = 10$$), this would be considered a very surprising result for a fair coin, but not surprising if the coin were (strongly?) biased. Note that the extent of bias is related to how surprising such an outcome might be; e.g., if the coin only shows heads $$51\%$$ of the time (weak bias), then getting $$X = 10$$ is still quite surprising, even though strictly speaking $$\theta \ne 0.5$$. This is a subtle but important fact. Here, the alternative hypothesis is simply "the coin is not fair"--it encompasses all kinds of possibilities for how unfair it is, and again, intuition suggests that it would be in some sense "more difficult" to test for a coin that might be biased only slightly, compared to a coin that is, say, two-headed. This is where the concept of statistical power arises. For a fixed experimental design--the $$n = 10$$ coin tosses--we can imagine that it would be very difficult to design a test that could reliably tell the difference between a coin that is perfectly fair, versus a coin that shows heads $$51\%$$ of the time, i.e. $$H_{10} : \theta = 0.5, \quad \text{vs.} \quad H_{1a} : \theta = 0.51,$$ but it would be relatively easy to design a test for $$H_{20} : \theta = 0.5, \quad \text{vs.} \quad H_{2a} : \theta = 0.99.$$ Let us study these in more detail. Suppose that the true state of the coin is that it is either exactly fair, or that it shows heads with probability $$0.51$$, and there are no other possibilities. Then a test statistic would be $$T = X/10$$, the proportion of heads out of $$10$$ tosses. What would be a suitable rejection region for $$T$$? Well, clearly if $$T \le 0.5$$, we can't reject $$H_{10}$$. If $$T \ge 0.6$$, this is greater than $$0.51$$, so we probably should reject $$H_{10}$$. Our experiment isn't very discriminating. So what is the Type I error? It is $$\alpha = \Pr[\text{Reject } H_{10} \mid H_{10} \text{ is true}] = \Pr[X \ge 6 \mid \theta = 0.5] = \sum_{k=6}^{10} \binom{10}{k} (0.5)^k (1 - 0.5)^{10-k} = \frac{193}{512} = 0.376953.$$ Not so good. What is the Type II error? It is $$\beta = \Pr[X \le 5 \mid \theta = 0.51] = \sum_{k=0}^5 \binom{10}{k} (0.51)^k (1-0.51)^{10-k} = 0.598205.$$ This is also enormous, but not surprising. Is it true that $$\alpha = 1-\beta$$? It's close, but not quite.

Could we have done better than this by choosing a different rejection region? Well, suppose we reject $$H_{10}$$ if $$X \ge x$$. Then $$\alpha = \sum_{k=x}^{10} \binom{10}{k} (0.5)^{10},$$ which is a decreasing function of $$x$$, so if we make the test harder to reject, our Type I error decreases, as expected. But this in turn increases the Type II error: $$\beta = \sum_{k=0}^{x-1} \binom{10}{k} (0.51)^k (0.49)^{10-k}$$ which is an increasing function of $$x$$. You can't decrease both simultaneously. For fun, here's a table for each decision $$x \in \{0, 1, 2, \ldots, 10\}$$: $$\begin{array}{c|c|c|c} x & \alpha & \beta & \alpha+\beta \\ \hline 0 & 1. & 0 & 1. \\ 1 & 0.999023 & 0.000797923 & 0.999821 \\ 2 & 0.989258 & 0.00910283 & 0.998361 \\ 3 & 0.945313 & 0.0480003 & 0.993313 \\ 4 & 0.828125 & 0.155961 & 0.984086 \\ 5 & 0.623047 & 0.352603 & 0.97565 \\ 6 & 0.376953 & 0.598205 & 0.975158 \\ 7 & 0.171875 & 0.811227 & 0.983102 \\ 8 & 0.0546875 & 0.937922 & 0.99261 \\ 9 & 0.0107422 & 0.987372 & 0.998114 \\ 10 & 0.000976563 & 0.99881 & 0.999786 \\ \end{array}$$ So, as I said at the beginning, is it possible for $$\alpha = 1-\beta$$? Sure, but does that make for a meaningful test, or is it true in general? No. You will also notice that our original choice of rejection region, $$X \ge 6$$, minimizes the sum of the errors, but it's still huge.

How about the second hypothesis? Remember, now we are dealing with a coin that is either fair, or it shows heads $$99\%$$ of the time (and the choice is between these two possibilities). Well, we just go through the same calculations: $$\alpha = \Pr[X \ge x \mid \theta = 0.5]$$ is exactly the same, but the Type II error calculation changes: $$\beta = \Pr[X < x \mid \theta = 0.99] = \sum_{k=0}^{x-1} \binom{10}{k} (0.99)^k (0.01)^{10-k},$$ and now the table looks like this: $$\begin{array}{c|c|c|c} x & \alpha & \beta & \alpha+\beta \\ \hline 0 & 1. & 0 & 1. \\ 1 & 0.999023 & 1. \times 10^{-20} & 0.999023 \\ 2 & 0.989258 & 9.91 \times 10^{-18} & 0.989258 \\ 3 & 0.945313 & 4.42036 \times 10^{-15} & 0.945313 \\ 4 & 0.828125 & 1.16878 \times 10^{-12} & 0.828125 \\ 5 & 0.623047 & 2.02894 \times 10^{-10} & 0.623047 \\ 6 & 0.376953 & 2.41678 \times 10^{-8} & 0.376953 \\ 7 & 0.171875 & 2.00128 \times 10^{-6} & 0.171877 \\ 8 & 0.0546875 & 0.000113849 & 0.0548013 \\ 9 & 0.0107422 & 0.0042662 & 0.0150084 \\ 10 & 0.000976563 & 0.0956179 & 0.0965945 \\ \end{array}$$ One thing to observe here is that in some sense, when the Type I and II errors are both small, you have a "good" test. So $$\alpha + \beta = 1$$ is not something you want to have in your test, but that should have been obvious from the start.

Based on this table, what would you choose for your rejection region? There isn't one perfect answer here. If you must minimize your Type I error, you'd probably want to choose the last row. If you want to have higher power than this choice, and are willing to trade off some Type I error to get it, you might want to pick $$x = 8$$ or $$x = 9$$.

• I had no other option than to accept your detailed and nice answer. Thank you! I'll reread it more times and I may have questions afterwards. – user122424 Nov 12 '18 at 14:42
• Why is it the case that you say: "Well, clearly if $T≤0.5$, we can't reject $H_{10}$." ? – user122424 Nov 29 '18 at 11:38

Suppose you have two boxes of dice, one is a box of fair dice in which all faces are equally likely. The other has loaded dice for which the probability of getting a six is is 1/3. The labels are missing so you will roll a sample of 50 dice from each box to try to identify which box has the loaded dice.

Let $$H_0: \text{FAIR},$$ so that $$p_0(6) = 1/6$$ and let $$H_a: \text{LOADED},$$ so that $$p_a(6) = 1/3.$$

In the figure below, blue bars represent the null distribution under which the number of 6's seen in $$n = 50$$ trials is $$\mathsf{Binom}(n = 50,\, p = 1/6).$$ And let the brown bars represent the alternative distribution under which the number of 6's seen is $$\mathsf{Binom}(n = 50,\, p = 1/3).$$

You choose critical value $$c = 10.5$$ (dotted line). Thus $$\alpha = P(S \ge 11 \,|\, p=1/6) = .2014,$$ and $$\beta = P(S \le 10 \,|\, p = 1/3) = .0284.$$ Thus the 'power' of the test is $$1 - \beta = P(\text{Rej } H_0 | H_a \text{ True}) = P(S \ge 11 | p=1/3) \\ = 1-.0284 = .9716.$$

sum(dbinom(11:50, 50, 1/6))
[1] 0.2013702
sum(dbinom(0:10, 50, 1/3))
[1] 0.02844031


Note: In many practical applications it seems reasonable to design an experiment so that the significance level $$\alpha$$ is approximately the same as the power $$1 - \beta.$$

Here, perhaps you chose to make them different because you think it would be more serious to sell a loaded die to a customer who wants a fair one, than to sell a fair die to someone in the market for a loaded one.

[If you wanted significance level and power to be more nearly equal, you could pick the critical value $$c$$ to be near the middle of the region where the two distributions 'overlap'.]

Let’s say we’re testing a coin with probability $$p$$ of heads and our null hypothesis is $$p=0$$ (that it never comes up heads. We flip 10 times and our procedure is that we reject if we see a head, otherwise we accept.

If the null hypothesis is true we will definitely not see a head, so $$\alpha=0.$$ However if not and we have $$p>0,$$ then the probability of 10 tails in a row is $$(1-p)^n,$$ so we have $$\beta =(1-p)^n\ne 1-\alpha.$$

There is no reason whatsoever to expect $$\alpha=1-\beta.$$ Usually, since the null hypothesis is a simple hypothesis, $$\alpha$$ is just a number, whereas since the alternative hypothesis is compound, $$\beta$$ depends on the effect size. So there’s usually a particular effect size for which they’re equal (in this silly example it’s $$p=0$$), but there is no real significance to this value. (One would hope they were both small for effect sizes of interest. Traditionally, the rule of thumb is to fix 5% for $$\alpha$$ and then set sample size so that $$\beta$$ is less than 20%.)

• Please, what do you mean by "effect size" ? – user122424 Nov 11 '18 at 19:56
• The effect size is the amount by which the true value of the parameter is different from the null. – spaceisdarkgreen Nov 11 '18 at 20:03
• May I ask you yet, how have you derived in your second paragraph $(1-p)^n\neq 1-\alpha$? – user122424 Nov 13 '18 at 15:39
• $p>0,$ so $(1-p)^n <1.$ $\alpha =0,$ so $1-\alpha=1.$ – spaceisdarkgreen Nov 13 '18 at 15:46