0
$\begingroup$

I have asked in this question on how to perform a classical hypothesis test to determine given the observation $x = 1$ which of the two probability models:

$$P_0(x) = \begin{cases} 0.009 & \text{if $x = 1$} \\ 0.001 & \text{if $x = 2$} \\ 0.990 & \text{if $x = 3$} \end{cases}$$

$$P_1(x) = \begin{cases} 0.001 & \text{if $x = 1$} \\ 0.989 & \text{if $x = 2$} \\ 0.010 & \text{if $x = 3$} \end{cases}$$

is more likely. It resulted that the classical paradigm would reject the null hypothesis at a confidence level of $\alpha = 0.01$ so that we must choose model $P_1$.

Then I'm asked to perform a Bayesian test on them with prior $\pi(H_0) = \pi(H_1) = 1/2$ where $H_i$ are the hypothesis of the test:

$$\begin{cases} H_0: & P_0(x) \\ H_1: & P_1(x) \end{cases}$$

However, with the Bayesian approach, the posterior odds is $Q^* = \frac{P(H_0|x = 1)}{P(H_0|x = 1)} = \frac{0.009}{0.001} = 9$ where I used Bayes theorem (conditional probability formula also works). This result is interpreted as $H_0$ being 9 times more likely than $H_1$.

My question

  1. Is my procedure correct?
  2. How can I interpret this difference between the two approaches? What test should I trust in this situation?
$\endgroup$

1 Answer 1

1
$\begingroup$

Your procedure is correct

The answer to a second question is: this two tests has different quality criteria, therefore their answers differ. The first one has maximal power under the fixed $\alpha$-level $\alpha=0.01$. The Bayesian test has a minimum weighted average error $$\pi(H_0)P(X \in R|P_0)+\pi(H_1)P(X \not\in R|P_1).$$

Therefore, these tests are not required to give the same result. And you can get the same results in this case if you reduce $\alpha$-level.

For $\alpha=0.001$ only $x=2$ appears in reject area: $R=\{2\}$ and no difference between the results of this two tests appears.

Your value of $\alpha$ is $\alpha=0.01$. This value is relatively large with respect to $0.001$. That is, you want to reject the null hypothesis not only for value $x=2$, which occurs with a probability $0.001$ under $H_0$, but additionally for some other value of the sample. The value $x=1$ is that extra value, because the likelihood ratio in it is the next largest. Therefore the point $x=1$ has fallen into reject area.

If you take $\alpha=1$, you will reject $H_0$ for any sample value: $R=\{1,2,3\}$. This is an extreme case.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .