Most of the time, to do hypothesis tests is to follow the tedious processes given on statistics textbooks. But recently I find that there are, in general, more than several test statistics that can be used to test the same hypothesis. To give an example consider the following test.

$n$ samples of the random variable $X$ are given. Null Hypothesis $H_0:X\sim N(\mu,\sigma_0^2)$. Alternative Hypothesis $H_1:X\sim N(\mu,\sigma_1^2)$. $\mu$ and $\sigma_i$ are known. $\sigma_0<\sigma_1$.

If we follow the textbooks, then we should use the $\chi$-square distribution, and the test statistic is the variance of the sample(times $n$ divided by $\sigma_0$). But the following test is also appropriate.

Let $\bar{X}$ be the sample mean. If $|\bar{X}-\mu|<c$ then accept $H_0$. Otherwise accept $H_1$. $c$ is chosen according to the level of significance of the test, $\alpha$.

This is an appropriate test because the condition $|\bar{X}-\mu|<c$ is more likely to be satisfied under $H_0$ than under $H_1$. (Many other test statistics are also appropriate for the same reason.) The test, however, has one shortcoming: the probability of making a type II error, $\beta$, is independent of $n$, which means it doesn't decrease as sample size increases.

Therefore I wonder which test statistic is the best in this situation. There is some simple criteria of how good a test statistic is. In this case, for example, we can say that for $n=100$ and $\alpha=0.05$, the smaller the minimum value of $\sigma_1/\sigma_0$ required to make $\beta\leq 0.05$, the better the test statistic.

Also, how can we make sure that the test given on the textbook is the best test, and there are no better alternatives?

Since this question might be too broad, I would be glad if anyone could give a specific example(like the one above) to illustrate the general idea. Also, are there any books or articles that discuss this topic in detail?


Typically, with a simple null and alternative hypothesis, the aim is to keep the probability of wrongly rejecting the null hypothesis (a Type I error) below a particular value (the significance level of the test) and then minimise the probability of wrongly failing to reject the null hypothesis (a Type II error), i.e. maximising the power of the test. Some tests are more powerful than others for the same level of significance

So let's take your suggested test statistics with some arbitrary specific numbers: $H_0: X_i \sim \mathcal N(100,4)$ against $H_1: X_i \sim \mathcal N(100,9)$ with a sample size of $10$ aiming for a significance level of $0.05$, and compare:

  • Test A based on $\frac1{\sigma^2} \sum\limits_i (X_i-\mu)^2 \sim \chi^2_n$: reject $H_0$ if $\frac1{4} \sum\limits_{i=1}^{10} (X_i-100)^2 \gt 18.307$. This has a probability of rejection of about $0.6155$ if $H_1$ is correct but $0.05$ if $H_0$ is correct

  • Test B based on $ \frac{\sqrt{n}}{\sigma}\left(\bar{X}-\mu\right) \sim \mathcal N(0,1)$: reject $H_0$ if $\frac{\sqrt{10}}{\sqrt{4}} \bigg|\bar{X}-100\bigg| \gt 1.96$. This has a probability of rejection of about $0.1913$ if $H_1$ is correct but $0.05$ if $H_0$ is correct

So Test A is more than three times as powerful as Test B in this particular case (seen another way, Test A is less than half as likely to make Type II errors), and in that sense Test A is a better test

With larger $n$ the distinction is even greater: with $n=100$ the power of Test A using $\chi^2_{100}$ rises to about $0.9999$ while, as you say, the power of Test B stays constant, at about $0.1913$

The Neyman–Pearson_lemma can point to the uniformly most powerful test for simple hypotheses


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.