2
$\begingroup$

I am currently learning about hypothesis testing and I really don't understand why do we take into consideration the probability of the events to the left and to the right of the observed value (or just to the left, or just to the right in the cases when we are interested if a parameter is only greater or smaller than the hypothesized value of the parameter). So it is clear enough that we take into consideration the probability of the observed value, but why do we also take into consideration the probability of the events that are more extreme than the observed value?

It seems to me that if we also take into consideration the events that are more extreme than the observed value, we are overestimating the p-value. I understand that we couldn't consider only the observed value if we are talking about a continuous distribution, since in that case we have to find the area under the curve to find the probability, and we would find the area of a line which would be $0$. But we could consider a small interval or something like that. And in the case of a discrete distribution we wouldn't have this problem, but we still take into consideration the events that are more extreme than the observed value.

So why does this work? I would really appreciate it if you could explain it like you would explain it to someone who is just starting with statistics, since that is the position that I am in.

$\endgroup$
  • $\begingroup$ Because hypotheses are always stated in the form that some random variable has value $x_0$ OR GREATER. $\endgroup$ – David G. Stork Apr 6 at 20:08
0
$\begingroup$

Let's take normal data modelled by the distribution $X \sim (\mu, \sigma^{2}) $ as our example. The distribution says the mean is $\mu$, whereas we take a sample from the data and that sample has mean $\bar{x} \neq \mu$. So, did we just get an unlucky sample, or is the model actually incorrect and we need to change the value of $\mu$?

The answer is, of course, just how different is $\bar{x}$ from $\mu$? If $\bar{x}$ is really close, that is a perfectly likely outcome. If $\bar{x}$ is very far from $\mu$, however, that is unlikely - we should be suspicious of $\mu$.

The thing is, we need to decide in advance our threshold for being suspicious enough of $\mu$ that we change the model. Say $5\%$. We want to find the range of values for $\bar{x}$ where the probability of getting that $\bar{x}$ is less than $5\%$. If we calculate a $\bar{x}$ and it falls in that range, then we say 'Hey! This is unlikely! I think $\mu$ needs to be changed!'.

I think you are considering calculating $\bar{x}$ first, and then figuring out the probability that we got that $\bar{x}$, and then drawing a conclusion afterwards, which is the wrong way around.

I'm going to give you an analogy I use with my students. Let's say a fellow comes up to you and he says he is a world champion at darts. Do you believe him? Well, let's assume he is not a world champion ($H_{0}$) and go and grab a dartboard. We should draw an area on that dartboard. If he can land a dart in that area, we will say he is a world champion. But we want to be sure so let's draw an area in the board which is $5\%$ of the total area of the board. If he gets the dart in there, we will reject our $H_{0}$ because he has done a thing which is not likely to happen at random.

You seem to be worrying about all of the other points in the area of the dartboard we drew around to challenge him. Well, just because he didn't hit all the other areas (he only had one dart!) it doesn't mean he hasn't proved himself. We had to draw the small area in order to challenge him. We couldn't give him a specific point to hit - that is impossible and would give us a false negative every time. Also, we can't just let him throw a dart and look where it landed and go 'huh - that wasn't likely, I guess he is a champion'. We have to set up the challenge first (choose a $p$ value) and then let him throw (calculate a $\bar{x}$).

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ "I think you are considering calculating x¯ first, and then figuring out the probability that we got that x¯, and then drawing a conclusion afterwards, which is the wrong way around." that's exactly the way I was thinking about it, but didn't know how to say it. Thanks a lot for clearing that. $\endgroup$ – user592938 Apr 6 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy