I am trying to figure out the best statistical test for a data set I've collected. I presented a group of adults (n=9) with a 21 different auditory cues. For each cue I asked them to choose the corresponding word from list of four words, one which was "correct" and three "incorrect." I then had them take the test again (I would have run more trials, but I didn't have enough time).

Mainly I want to test if their answers show that they recognized the cue and assigned meaning to it, or if they chose it by chance. But I'm getting caught up in the details and it has been a while since I took a stats course.

I want to compare the number of correct choices to the theoretical amount of correct choices (1/4). Should I use a binomial test or Chi-squared? Or do you recommend something totally different? Also should I look at each patient's individual data or the aggregate?

I also wanted to analyze the amount "recognized" for each patient - the amount of times the patient chose the same word for both the test and retest (despite if it was right or wrong). This is where I am really lost - could I simply compare the total "recognized" to the expected 1/4? (I got 1/4 by looking at the probability of selecting the correct twice [1/4*1/4=1/16] and selecting the same wrong twice [3/4*1/4= 3/16])?

I'm sure there is a simple solution, but I may be overthinking this. Any help is much appreciated!

  • $\begingroup$ I would suggest the binomial distribution. Assume, the answers were chosen by chance, then the answer is correct with probability $\frac{1}{4}$. Calculate, how likely the result is under this assumption. $\endgroup$ – Peter Sep 8 '15 at 21:51

Suppose you want to take both the test and re-test into account.

On the first test a subject who is just guessing performs $r = 21$ trials with success probability $p = 1/4$. Again on the re-test the subject does the same. So the subject's total score is on the two tests is a binomial random variable $X$ with $t = 2n = 42$ trials and success probability $p = 1/4$ on each. Thus the average (or expected) value is $\mu = E(X) = tp = 42/4 = 10.5,$ variance $V(X) = tp(1-p) = 126/16 = 7.875$, and $\sigma = SD(X) = \sqrt{7.875} = 2.8062.$ Of course, we expect that subjects who are not just guessing will tend to get higher scores.

For $n = 9$ subjects you would find the average score $\bar X.$ Then $$Var(\bar X) = \sigma^2/n = 7.875/9 = 0.875$$ and $SD(\bar X) = 0.9354.$

Your null hypothesis is $H_0: \mu = 10.5$ and your alternative hypothesis is that $H_a: \mu > 10.5.$ The average score $\bar X$ is close enough to normal that you can reject the null hypothesis at the 5% level if $Z = (\bar X - 10.5)/0.9354 \ge 1.645$ or $\bar X \ge 12.$

Notes: (a) This is assuming that the 42 trials for each subject are independent, this includes the assumption that the first test and re-test are independent. (b) We also assume that the nine subjects are chosen at random for the population of interest. (c) Because the standard deviation of $\bar X$ under the null hypothesis is know, this is a Z-test. (d) Similar tests could be done for individual subject (use his/her $X$ instead of $\bar X$ with the appropriate $\sigma$). Also, you could do a similar procedure for the first test only (use $n = 21$ instead of $t = 42$). (e) Just $X$ for a given subject is nearly normal, so there is no problem assuming that $\bar X$ is normal.

  • $\begingroup$ Great - that makes sense, just to make sure I've got this now. I could also test the "re-test reliability" of sorts by analyzing how many times a subject chose the same answer on the test and then the retest. Here r=21 trials and p=1/4 (The probability of correct twice (1/16) and the probability or the same wrong twice (3/16)). Find X (for one subject) and calculate the Z-score. Is that right? $\endgroup$ – user269073 Sep 9 '15 at 17:32
  • $\begingroup$ Another matter entirely: Are words and cues the same between the two test sessions? Is there likely to be a memory or learning component in results on 2nd test? Null distribution assuming everything is a guess is much easier. You can test memory/learning if questions same: chance a person replicates exact result at random is 1/4. You could check if each person's results are consistent with that: BINOM(21, 1/4). $\endgroup$ – BruceET Sep 9 '15 at 18:32
  • $\begingroup$ Seems to me that in comparing two sessions, it is only matching answers that matter (whether right or wrong). What relevant info is there for $comparison$ in RW or WR pairs? Maybe you have an objective in mind that I don't see. But this is an additional topic, maybe best handled in a new Question. $\endgroup$ – BruceET Sep 9 '15 at 20:03
  • $\begingroup$ There is likely to be a learning component and that is what I am trying to identify by comparing the two tests. The cues and questions are the same so I have been comparing the chance of replication using BINOM. I'm not sure if that gets to your question, but here's what I've done so far - if it clears anything up.Thus far I've found the p-value for all the successes from the entire data sat (n=42*9=378), the p-value for the first and second test separately (n=21*9=189) and the p-value for the retest (n=189).... $\endgroup$ – user269073 Sep 10 '15 at 18:09
  • $\begingroup$ I feel that analyzing test 1 and 2 separately makes sense, since they are not truly independent. I also looked at the p-value for the amount correct per subject as well to see variation in user, but I'm not sure if this p-value has any real meaning to it, or if just the value for the entire data set is worth presenting. $\endgroup$ – user269073 Sep 10 '15 at 18:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.