hypothesis testing True or False

Suppose a test of $H_0: μ =0$ against $H_1: μ \ne 0$ resulted in rejection of $H_0$ at the 5 percent level of significance.

$1,$ When we reject $H_0,$ we know $H_1$ is true, but we don't know if $μ$ is far away from $0,$ right? I can't conclude that $μ$ is far away from $0,$ right?

$2.$ Also, can I say: The probability that $μ$ is equal to $0$ is less than 0.05? Because rejecting $H_0$ at 5% significance level means P-value is smaller than 0.05?

Rejecting $H_0: \mu = 0$ is a statement about the data. The data are not consistent with drawing from a population with mean $0.$

$1.$ Strictly speaking, rejecting $H_0$ does not mean $H_1$ is true. The sample mean $\bar X$ must have been relatively 'far' from $0$ as measured by the test statistic.

For example, suppose you have a random sample of size $n = 16$ from a normal population with $\sigma = 12$ and sample mean $\bar X = 6.75.$ Then the test statistic is $Z = \frac{\bar X - 0}{\sigma/\sqrt{n}} = 6.75/3 = 2.25.$ You reject $H_0$ against $H_1$ at the 5% level when $|Z| > 1.96.$ And here $|Z| = 2.25 > 1.96,$ so we reject $H_0.$

While rejecting $H_0$ does not say anything specific about how far $\mu$ is from $0,$ the information used in the test of this hypothesis allows you to make a 95% confidence interval for $\mu:$ $$\bar X \pm 1.96\sigma/\sqrt{n}.$$ In my example above, this computes to the interval $6.75 \pm 5.88$ or $(0.87, 12.63),$ which gives some information about how far from $0$ the population mean $\mu$ must lie.

The truth is that $\mu$ either lies in this interval or not. In this case it seems best to act as if $\mu \ne 0$ because 0 does not lie in the interval. If you go through life believing such intervals are correct, you will be right 95% of the time; this time--probably Yes, maybe No.

$2.$ You cannot use the significance level 5% = 0.05 to make probability statements about $\mu.$ The significance level is the probability your decision to reject $H_0$ was wrong.

In my example, assuming $\mu = 0,$ the P-value is the probability of seeing a value of the sample mean that makes $|Z| > 2.25.$ That computes to a P-value of 0.024, which is smaller than 0.05. But $\bar X$ was used in computing $Z$, so the P-value amounts to another probability statement about the sample mean $\bar X,$ not about the population mean $\mu.$

In summary, you need to realize that statistical testing is an inductive process, not a deductive one. In many parts of mathematics, you are given axioms, from which you deduce other statements using logic. Statistics uses mathematics, but not in exactly in that way for interpreting results of hypothesis tests and confidence intervals.

By contrast, in statistics we begin with data and perhaps some assumptions about the population from which the data came. We can say, "It is very unlikely your 16 observations are a random sample from a normal population with mean 0 and SD 12." It is up to you to decide to take the hint that $\mu \ne 0$ or to believe that an event of probability 0.024 has occurred and insist that $\mu = 0$ in spite of the evidence from your 16 observations. We used mathematics to find the probability 0.024, but we can't make true or false statements whether nor not $\mu = 0.$

Statisticians make statements in very careful ways, trying to make statements about data that help people make decisions about the populations the data are sampled from. Some of the statements about hypothesis testing and confidence intervals may seem vague and convoluted. These statements are intended to be helpful in making decisions, but it takes some practice getting used to them.

It may help you to realize that statisticians see only the data and not the whole populations. They can make valid statements about the data, but only help you make informed guesses about populations.