I am confused about the notion of the significance level ($\alpha$) in hypothesis testing. There seems to be two (at least) ways of looking at the significance level, and I am unable to reconcile them, that is, I don't understand how they are equivalent. From this article, the author first states:

The significance level, also denoted as alpha or α, is the probability of rejecting the null hypothesis when it is true.

Then he shows this normal distribution and goes on:

The critical region [shown in red on the provided image] defines how far away our sample statistic must be from the null hypothesis value before we can say it is unusual enough to reject the null hypothesis.

Why is the concept of choosing a threshold for the hypothesis test related to the concept of having a false positive?


Even when the null hypothesis is true, there's is always a nonzero probability just by chance (due to randomness in the sampling procedure etc) to observe something very different from the typical behavior (far away from the average). As you already understood, this is the probability of false positive.

Because there's always this nonzero chance, mathematically speaking, seeing something unusual is never a "proof" that the null is false.

However, in reality, when something theoretically too unlikely has actually happened, our common sense dictates that we would rather reject the original theory instead of just saying "oh well, weird things can happen just by chance so this doesn't mean anything."

In other words, coincidences indeed happen from time to time and they often provide no information. Nonetheless, when the data seems too implausible, we would rather think that there's some reason behind it.

"How unlikely is too unlikely" is the level $\alpha$, which you should determine before the data is collected. The some reason behind the observed data is the alternative theory (hypothesis) we'd like to take instead of the original theory (null hypothesis).


Referring to the figure linked to the question, if we would reject the null hypothesis when the sample statistic is $330,$ then surely we should also reject it when the statistic is $331,$ or $340,$ or $350,$ or $1734.51.$

That's why you have a critical region. Once you've decided that there are certain values of the statistic that are so extreme that you should reject the null hypothesis, then all of the more extreme values--anything in the red regions--also causes rejection.

I did not see any use of the term "false positive" in the linked article, but I suppose that when that term occurs in the question, it means the event that the null hypothesis actually was true but we decided to reject it.

Because the bell curve in the figure is the probability distribution of a random sample, assuming that the null hypothesis is true, the area of the rejection region is the probability that (just by bad luck) you get a sample statistic that is in the rejection region (that is, you get a false positive result), given that the null hypothesis actually is true.

In other words, the area of the rejection region is precisely the probability of rejecting the null hypothesis when it is true.

Therefore the area of the rejection region is equal to the significance level, $\alpha,$ and we must choose the threshold so that $\alpha$ is a small enough number for our results to be considered significant.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.