# Is p value sufficient to compare the significance

What we have is the following:

Three groups of students are taking the same pre-test and post-test on a Math subject. In between they get to play with tutoring system A (for first group), tutoring system B (for second group), and tutoring system C (for last group). Here are the results:

Group A: Pre-test Post-tet

Student 1a 20% 34%

Student 2a 32% 45%

Student 3a 12% 40%

Student 4a 52% 55%

Student 5a 37% 39%

Group B: Pre-test Post-tet

Student 1b 16% 31%

Student 2b 22% 35%

Student 3b 32% 40%

Student 4b 25% 31%

Student 5b 27% 31%

Group B: Pre-test Post-tet

Student 1b 33% 51%

Student 2b 12% 25%

Student 3b 42% 45%

Student 4b 39% 45%

Student 5b 17% 33%

Question (1): Would it be sufficient to run a paired t-test for each group and use the p values to compare the effectiveness of three systems.

Question (2): If not, what additional test that I can run to make the comparison more convincing

Doing three paired t tests would not tell you what you want to know. Each paired test would tell you whether students in one of the three groups (tutorial systems) shows improvement. The answer to that is Yes for all groups, because each group shows all five students with positive improvement scores ($1/2^5 < .05$). But this says nothing at all about the relative effectiveness of the three groups.

One appropriate procedure would be to find an improvement score for each person 14 for student 1a. Then do a one-factor (one-way) ANOVA on the three sets of improvement scores. If the F-test for the ANOVA shows significance (i.e., that not all three tutorial systems are equally effective), then you would need to do some kind of multiple comparison procedure to see what pattern of difference is plausible.

A complication is that these are percentages and so not really normally distributed. There are several possibilities if you suspect (or find from a test on residuals) that data are far from normal: Kruskal-Wallis and permutation tests are popular alternatives.

The title of the Question asks whether the P-value is sufficiently small to declare significant differences. But you don't show a P-value, so that it not a good title for this Question.

I put these data into Minitab statistical software (quickly and without proofreading, better check). The resulting ANOVA is as follows:

 Source  DF     SS    MS     F      P
Group    2   26.1  13.1  0.42  0.668
Error   12  375.6  31.3
Total   14  401.7


The P-value shown is not sufficient to reject the null hypothesis that improvement scores are the same for all three tutorial programs. I don't know whether you have studied ANOVA or not. If not, I wonder about the purpose of the question. If so, I suppose you will know how to interpret the ANOVA table.

The largest difference among the three sample means is a little over 3 (not shown). This ANOVA procedure assumes that the variances are the same in the three groups. It estimates the common group variance to be about 31 (Ms for Error).

Note: When significance is not found, it is good statistical practice to do power computations to see whether the study had any chance of success in detecting differences among group population means.

(a) A one-way ANOVA with 3 groups, 5 replications within each group, and a common variance of 31 has only about one chance in ten of detecting a difference of 3 in improvement scores, even if it is real.

(b) It is difficult to say how large a difference investigators would find interesting. In order reliably to detect a difference of 5 points between the best and worst tutorial system one would need over 30 students in each group.

• I have not thought about the ANOVA, and the purpose of the question is (maybe you can just answer my question first then we can go from there), there will be three p values for three separate paired t test (one for each group of scores) and can I compare these three p-values to draw a conclusion which is whichever is smallest is the most effective and vice versa. Aug 22 '15 at 3:26
• That might work if two of the paired t-tests were clearly not significant and one is clearly significant, but that is not the case here. How much difference among P-values would it take for you to be sure you've found a meaningful difference among groups? Not a stupid idea, but it doesn't work at all here, and has problems overall. I did 'just answer your question (1)' above and with more detail now. For your question (2), you need to start thinking about ANOVA, now seems a good time. Aug 22 '15 at 3:35
• Sure, I just saw it. Aug 22 '15 at 3:43