3
$\begingroup$

A while back I've asked a question on the relationship of the total variation distance between probability measures to hypothesis testing and got a very nice answer. I understand that that answer gives a trade-off relationship between the probability of type I error (false positive) $\alpha$ and a type II error (miss) $\beta$, similar to what Neyman-Pearson lemma provides.

Within the Neyman-Pearson framework, one can set $\alpha$ arbitrarily close to 0 at the expense of the power of the statistical test $1-\beta$, however, as far as I understand, one can not set $\alpha=0$.

I am wondering if there are non-trivial hypothesis tests out there allow one to set $\alpha=0$. I haven't encountered one in my reading. My intuition tells me that there aren't because 1) a hypothesis test must be a threshold-based test; and 2) as long as the probability distributions associated with the hypotheses are different, any non-trivial threshold test (i.e. a test that doesn't always accept the null hypothesis) has some finite chance of falsely rejecting the null hypothesis.

However, I thought I'd ask the experts here whether my intuition, and the reasoning behind this intuition, is correct. Perhaps there are statistical hypothesis tests not based on thresholds out there...

$\endgroup$
4
$\begingroup$

I am wondering if there are non-trivial hypothesis tests out there allow one to set $\alpha=0$. I haven't encountered one in my reading. My intuition tells me that there aren't because 1) a hypothesis test must be a threshold-based test; and 2) as long as the probability distributions associated with the hypotheses are different, any non-trivial threshold test (i.e. a test that doesn't always accept the null hypothesis) has some finite chance of falsely rejecting the null hypothesis.

I suppose the answer depends on how far your definition of triviality extends.

Let the observation be $X$ where $X$ is a Bernoulli random variable whose parameter $p$ has value $\frac{1}{2}$ when $H_1$ is true and value $1$ when $H_0$ is true (i.e. $X$ is a degenerate random variable that equals $1$ with probability $1$ when $H_0$ is true). Thus the distributions are different. The likelihood ratio is $$\Lambda(X) = \frac{p_1(X)}{p_0(X)} = \begin{cases}\infty, & X = 0,\\0.5, & X = 1,\end{cases}$$ and so the maximum-likelihood decision rule (a threshold test with threshold $1$) chooses $H_1$ when $X = 0$, and chooses $H_0$ when $X = 1$. The false alarm probability $\alpha$ is $$\alpha = P\{H_1 ~\text{chosen}\mid H_0~\text{true}\} = P\{X = 0 \mid H_0~\text{true}\} = 0$$ while the power $1-\beta$ is $$1-\beta = P\{H_1 ~\text{chosen}\mid H_1~\text{true}\} = P\{X = 0 \mid H_1~\text{true}\} = \frac{1}{2}.$$

$\endgroup$
  • $\begingroup$ I like this example. Now, if I extend the definition of non-triviality to the hypotheses (i.e. all the subsets of the set of possible observations must have non-zero probabilities under each hypothesis), I think my intuition is right... and I think the proof is pretty easy. $\endgroup$ – M.B.M. Nov 29 '11 at 1:07
  • $\begingroup$ @Bullmoose You might want to phrase "all subsets..must have nonzero probabilities" a bit more carefully to avoid issues with measurability of arbitrary sets of real numbers, and with continuous random variables where $P\{X = a\} = 0$ for all $a$. $\endgroup$ – Dilip Sarwate Nov 29 '11 at 1:25
  • $\begingroup$ Yeah, thanks for noting that. Of course I meant Borel sets on $\sigma$-algebra. :) $\endgroup$ – M.B.M. Nov 29 '11 at 5:37
  • $\begingroup$ @Bullmoose Perhaps you need to state it as something like "the likelihood ratio is finite with probability $1$" rather than in terms of Borel sets and $\sigma$-algebras $\endgroup$ – Dilip Sarwate Nov 29 '11 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.