# Is hypothesis testing at $\alpha=0$ possible?

A while back I've asked a question on the relationship of the total variation distance between probability measures to hypothesis testing and got a very nice answer. I understand that that answer gives a trade-off relationship between the probability of type I error (false positive) $\alpha$ and a type II error (miss) $\beta$, similar to what Neyman-Pearson lemma provides.

Within the Neyman-Pearson framework, one can set $\alpha$ arbitrarily close to 0 at the expense of the power of the statistical test $1-\beta$, however, as far as I understand, one can not set $\alpha=0$.

I am wondering if there are non-trivial hypothesis tests out there allow one to set $\alpha=0$. I haven't encountered one in my reading. My intuition tells me that there aren't because 1) a hypothesis test must be a threshold-based test; and 2) as long as the probability distributions associated with the hypotheses are different, any non-trivial threshold test (i.e. a test that doesn't always accept the null hypothesis) has some finite chance of falsely rejecting the null hypothesis.

However, I thought I'd ask the experts here whether my intuition, and the reasoning behind this intuition, is correct. Perhaps there are statistical hypothesis tests not based on thresholds out there...

-

I am wondering if there are non-trivial hypothesis tests out there allow one to set $\alpha=0$. I haven't encountered one in my reading. My intuition tells me that there aren't because 1) a hypothesis test must be a threshold-based test; and 2) as long as the probability distributions associated with the hypotheses are different, any non-trivial threshold test (i.e. a test that doesn't always accept the null hypothesis) has some finite chance of falsely rejecting the null hypothesis.
Let the observation be $X$ where $X$ is a Bernoulli random variable whose parameter $p$ has value $\frac{1}{2}$ when $H_1$ is true and value $1$ when $H_0$ is true (i.e. $X$ is a degenerate random variable that equals $1$ with probability $1$ when $H_0$ is true). Thus the distributions are different. The likelihood ratio is $$\Lambda(X) = \frac{p_1(X)}{p_0(X)} = \begin{cases}\infty, & X = 0,\\0.5, & X = 1,\end{cases}$$ and so the maximum-likelihood decision rule (a threshold test with threshold $1$) chooses $H_1$ when $X = 0$, and chooses $H_0$ when $X = 1$. The false alarm probability $\alpha$ is $$\alpha = P\{H_1 ~\text{chosen}\mid H_0~\text{true}\} = P\{X = 0 \mid H_0~\text{true}\} = 0$$ while the power $\beta$ is $$\beta = P\{H_1 ~\text{chosen}\mid H_1~\text{true}\} = P\{X = 0 \mid H_1~\text{true}\} = \frac{1}{2}.$$
@Bullmoose You might want to phrase "all subsets..must have nonzero probabilities" a bit more carefully to avoid issues with measurability of arbitrary sets of real numbers, and with continuous random variables where $P\{X = a\} = 0$ for all $a$. – Dilip Sarwate Nov 29 '11 at 1:25
Yeah, thanks for noting that. Of course I meant Borel sets on $\sigma$-algebra. :) – M.B.M. Nov 29 '11 at 5:37
@Bullmoose Perhaps you need to state it as something like "the likelihood ratio is finite with probability $1$" rather than in terms of Borel sets and $\sigma$-algebras – Dilip Sarwate Nov 29 '11 at 14:57