A test was given to two sets of students, CONTROL and EXPERIMENT, that had question A and question B. I want to know if students who got question A right were more likely to get question B right, and I want to know if being in the CONTROL or EXPERIMENT groups made a difference in this relationship.

I know I can't use ANOVA, because the data results are CORRECT or INCORRECT, not normally distributed data. I thought maybe I could use Pearson's Chi-Squared test, but that only seems to deal with one factor at a time. I looked at the Cochran–Mantel–Haenszel test (the example given here: http://udel.edu/~mcdonald/statcmh.html ) but I can't seem to figure out if I can make it fit my question.

This has to be a common question. I'm trying to help someone who conducted an experiment with this, but I have very limited stats experience. Any help is appreciated.


This sounds like a good case for using Fisher's Exact Test. It sounds like you could assign your data into a 2x2 contingecy table. If that is the case, I would read up on the Tea Drinker Exp w/r Fisher's Test.

           Question A:
Question B Control Treatment Totals
Control     2      4         6
Treatment   5      6         11
Totals      7      10        17

p-value = R1!*R2!*C1!*C2! / { (grand total)! times (factorials of internals) }

p-value = (6!*11!*7!*10!) / (17!*2!*4!*5!*6!)

One more thing, MH test is a more specific example of the the Fisher's Test when used with multiple confounding levels, for example: 2x2xk matrices. Also I would not rule out the chi-square test but it assumes that you data is normal and this can be done but the warning is that you must have a large data set for that assumption of normality. The Fisher's Exact Test does not depend on the assumption of normality, ie good for small data sets.

  • $\begingroup$ Thanks! I'll take a look. $\endgroup$ – mpettis Mar 26 '13 at 1:03
  • $\begingroup$ As I thought about the other day yes you could chi-square if you discordant pairs are greater than 10. $\endgroup$ – oaxacamatt Mar 26 '13 at 13:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.