p-value, intuition about type-I error=$\alpha$ [duplicate]

In hypothesis testing consider this situation:

Define $\alpha=P\{\text{type I error}\}=P\{\text{ Rejecting } { H_0}\text{ when } H_0\text{ is true}\}$.

Also define $p-$value like this

What is the probability of observing a data which is similar to the one on hand, or more extreme, if $H_0$ happens to be true. This probability will be called the $p-$value.

How do I get some inutition about this: A test with a small $p-$value indicates that the null hypothesis is less plausible than the alternative hypothesis and in this case $H_0$ is rejected.

marked as duplicate by Aaron Montgomery, Chris Custer, Namaste, Dando18, Xander HendersonMay 5 '18 at 1:10

• Extreme here means an unlikely result which against $H_0$, so if the data/test statistics is more extreme, then you are more likely to reject $H_0$. $p$-value means, if you repeat the experiment again independently, what is the probability of obtaining a stronger evidence against $H_0$. If it is small, that mean it is unlikely to find a stronger evidence than the current one, in other words, the current evidence is strong enough to persuade you to reject $H_0$. – BGM May 4 '18 at 15:31
• In defining the p-value it is important to say that it is the probability under $H_0$ of obtaining a test statistic that is more extreme in the direction(s) of the alternative than what is observed. That way you get the correct inequality for a one-sided test and you get probabilities in two-tails for a two-sided test. – BruceET May 4 '18 at 16:22
• @BruceET But can you explain in some detail how the direction of the alternative corresponds to this probability: $\alpha=P\{\text{type I error}\}=P\{\text{ Rejecting } { H_0}\text{ when } H_0\text{ is true}\}$ ? Suppose that $H_1:\mu<\mu_0$ . – user122424 May 4 '18 at 17:58
• Perhaps best explained by examples. One recent relevant discussion here – BruceET May 4 '18 at 19:53

Two-sided test. Suppose you have the following ten observations from a normal population, and do a two -sided test $H_0: \mu = 100$ against the two sided alternative $H_1: \mu \ne 100.$

 102, 114, 114,  80,  83,  79, 117,  79, 115,  79


In R statistical software, the test looks like this:

t.test(x, mu=100)

One Sample t-test

data:  x
t = -0.68402, df = 9, p-value = 0.5112
alternative hypothesis: true mean is not equal to 100
sample estimates:
mean of x
96.2


Under the null hypothesis, the test statistic $T$ has Student's t distribution with 9 degrees of freedom. The observed P-value would be the probability of getting a value from that distribution that is less than $-0.684$ or is greater than $0.684,$ which is just as far away from 100, but in a positive direction. [In R, a CDF of Student's t distribution is denoted pt.]

 pt(-0.684, 9) + (1 - pt(0.684, 9))
## 0.5112009


In the figure below, you want the sum of the two probabilities (areas) that lie outside the vertical broken lines. One-sided test. Now suppose you test $H_0: \mu = 100$ against the one-sided alternative $H_0: \mu < 100.$ (Same data as above.)

        One Sample t-test

data:  x
t = -0.68402, df = 9, p-value = 0.2556
alternative hypothesis: true mean is less than 100
sample estimates:
mean of x
96.2


Now the P-value is only the probability in the left tail of the Student's t distribution, to the left of $-.684.$ (Because of the symmetry of the t distribution, this turns out to be half as large as the P-value for the two-sided test.)

pt(-0.684, 9)
## 0.2556005


In the figure, the P-value is only the probability to the left of the vertical broken line. Note about 'wrong sided' tests: If you had been unfortunate enough to choose the 'wrong' side in framing the alternative, testing $H_0: \mu = 100$ against the one-sided alternative $H_0: \mu > 100,$ then the P-value is the area to the right of the vertical broken line in the second figure. Key lines from the output of the t test in R would be as follows:

t = -0.68402, df = 9, p-value = 0.7444
alternative hypothesis: true mean is greater than 100


And using the CDF of Student's t distribution:

1 - pt(-.684, 9)
## 0.7443995