# How to determine which significance level to use?

I am doing a two sample hypothesis problem that goes like this:

Research has shown that good hip range of motion and strength in throwing athletes results in improved performance and decreased body stress. The article “Functional Hip Characteristics of Baseball Pitchers and Position Players” (Am. J. Sport. Med., 2010: 383–388) reported on a study involving samples of 40 professional pitchers and 40 professional position players. For the pitchers, the sample mean trail leg total arc of motion (degrees) was 75.6 with a sample standard deviation of 5.9, whereas the sample mean and sample standard deviation for position players were 79.6 and 7.6, respectively. Assuming normality, test appropriate hypotheses to decide whether true average range of motion for the pitchers is less than that for the position players (as hypothesized by the investigators).

Using the information from the question I was able to get a t value of $-2.63$, from what I understand from here I'm supposed to compare this -2.63 to a critical value I get from the t table, or I can get find the p-value (I got $0.0043$ from the normal table) and compare it to alpha which is the significance level. For both of these I don't really understand on which situations I'm supposed to be using for which method, It would be great if someone could explain this to me too. But regardless the main issue I have is that both of these requires a significance level to figure out, which wasn't given to me. Since I didn't know how to proceed I went to look at the answer to this question, which said this:

Because the one-tailed P-value is $.005 < .01$, conclude at the $.01$ level that the difference is as stated.

The alternative hypothesis was chosen because the P value was lower than the significance level of $.01$ but if they had just chosen a smaller significance level(e.g. $0.001$), then the alternative hypothesis would have been rejected

So here my question is how did they know that a significance level of $.01$ was the level they are supposed to compare to? Did they just choose it because there was no significance level given in the question?

## 2 Answers

As is somewhat usual in medical and physical therapy papers, the authors are not particularly well versed in the use of statistics. In this case, aside from the mistake of attempting to quote a two-sided P-value to test a one-sided hypotheses, and the mistacke of rounding $0.0043$ to $0.005$, they have done the ubiquitous but unjustifiable trick of selecting their significance criterion after finding the P-value from their data.

Since they would have reported an effect if the p-value came out to anything below $0.05$ and they assumedly would have made the same 1-sided/tow-sided error so they were actually testing for significance at the $0.025$ confidence level, an honest assessment is that they have demonstrated significance at the $0.025$ confidence level.

On the other hand, if they can honestly say they would not have published if their p-value came out to about $0.01$, then the data does justify claiming that level of confidence.

The answer is: it depends.

Most scientific publications require an $\alpha$ of at most .05. This seems to be mostly tradition; there is nothing distinctive about rejecting a true null hypothesis1 in 20 times. One does, however, have to consider practical aspects such as sample size an effect size when setting a confidence level.