Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

A while back I've asked a question on the relationship of the total variation distance between probability measures to hypothesis testing and got a very nice answer. I understand that that answer gives a trade-off relationship between the probability of type I error (false positive) $\alpha$ and a type II error (miss) $\beta$, similar to what Neyman-Pearson lemma provides.

Within the Neyman-Pearson framework, one can set $\alpha$ arbitrarily close to 0 at the expense of the power of the statistical test $1-\beta$, however, as far as I understand, one can not set $\alpha=0$.

I am wondering if there are non-trivial hypothesis tests out there allow one to set $\alpha=0$. I haven't encountered one in my reading. My intuition tells me that there aren't because 1) a hypothesis test must be a threshold-based test; and 2) as long as the probability distributions associated with the hypotheses are different, any non-trivial threshold test (i.e. a test that doesn't always accept the null hypothesis) has some finite chance of falsely rejecting the null hypothesis.

However, I thought I'd ask the experts here whether my intuition, and the reasoning behind this intuition, is correct. Perhaps there are statistical hypothesis tests not based on thresholds out there...

share|improve this question

1 Answer 1

up vote 3 down vote accepted

I am wondering if there are non-trivial hypothesis tests out there allow one to set $\alpha=0$. I haven't encountered one in my reading. My intuition tells me that there aren't because 1) a hypothesis test must be a threshold-based test; and 2) as long as the probability distributions associated with the hypotheses are different, any non-trivial threshold test (i.e. a test that doesn't always accept the null hypothesis) has some finite chance of falsely rejecting the null hypothesis.

I suppose the answer depends on how far your definition of triviality extends.

Let the observation be $X$ where $X$ is a Bernoulli random variable whose parameter $p$ has value $\frac{1}{2}$ when $H_1$ is true and value $1$ when $H_0$ is true (i.e. $X$ is a degenerate random variable that equals $1$ with probability $1$ when $H_0$ is true). Thus the distributions are different. The likelihood ratio is $$\Lambda(X) = \frac{p_1(X)}{p_0(X)} = \begin{cases}\infty, & X = 0,\\0.5, & X = 1,\end{cases}$$ and so the maximum-likelihood decision rule (a threshold test with threshold $1$) chooses $H_1$ when $X = 0$, and chooses $H_0$ when $X = 1$. The false alarm probability $\alpha$ is $$\alpha = P\{H_1 ~\text{chosen}\mid H_0~\text{true}\} = P\{X = 0 \mid H_0~\text{true}\} = 0$$ while the power $\beta$ is $$\beta = P\{H_1 ~\text{chosen}\mid H_1~\text{true}\} = P\{X = 0 \mid H_1~\text{true}\} = \frac{1}{2}.$$

share|improve this answer
    
I like this example. Now, if I extend the definition of non-triviality to the hypotheses (i.e. all the subsets of the set of possible observations must have non-zero probabilities under each hypothesis), I think my intuition is right... and I think the proof is pretty easy. –  M.B.M. Nov 29 '11 at 1:07
    
@Bullmoose You might want to phrase "all subsets..must have nonzero probabilities" a bit more carefully to avoid issues with measurability of arbitrary sets of real numbers, and with continuous random variables where $P\{X = a\} = 0$ for all $a$. –  Dilip Sarwate Nov 29 '11 at 1:25
    
Yeah, thanks for noting that. Of course I meant Borel sets on $\sigma$-algebra. :) –  M.B.M. Nov 29 '11 at 5:37
    
@Bullmoose Perhaps you need to state it as something like "the likelihood ratio is finite with probability $1$" rather than in terms of Borel sets and $\sigma$-algebras –  Dilip Sarwate Nov 29 '11 at 14:57

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.