2
$\begingroup$

I'm currently performing a study in which participants are being asked to listen to a continuous, changing musical tone and to answer when they detect a change and what they think that change correlates to (based off of prior training). The problem is that it's possible for them to answer more times than there are included changes in the sound clip. Is there an established method for scoring a test like this?

Example: There are 20 changes in the tone and the participant selects that 24 changes happened, but they got 14/20 of them correct.

I was arbitrarily thinking that their score should be #correct/20 if they provide 20 or less answers, but if they provide more than 20 answers, I was thinking it should be #correct/(20 + extra answers). Does this have any basis in research or should they be scored in a different manner?

$\endgroup$
2
$\begingroup$

Depends on how you want to “value” the mistake of hearing a change when one doesn’t exist. Your method has the advantage of bounding the test score by 0 below, i.e. no matter how many changes they get correct, the score is no smaller than. 0, and it’s only zero when they get none of the actual changes correct. Unfortunately the amount a person is dinged for an incorrect belief that a change occurred will depend on how many other times they make the same mistake, and it will also change the value of a correct answer. For example, if you get all 20 tones right but you thought there were 21, your score is 20/21 = 95.2, so you lost 4.8% for this wrong guess. On the other hand, if you only had 10 tones correctly identified, your score would be 10/21 = 47.6, which is a penalty of 2.4% for the same mistake. Similarly, even for a fixed number of correct answers, the more times you incorrectly think you hear a change, the less each wrong answer gives. The person who gets all 20 changes correct and answers 5 incorrect additional changes would get 20/25 = 80%, which is like 4% penalty for each wrong answer - but we saw above that the first wrong answer is a penalty of 4.8%.

This is really about how much you think making this kind of mistake should be worth compared to a correct answer. One option is to value incorrect observations of a change count as much as an correct answer, so you get +1 for each correct answer and -1 for each incorrect answer that is not a change in tone (and +0 for incorrectly identifying an actual tone). The final score could of course be negative this way.

Yet a third option is to ignore such attempts as simply irrelevant to the question you are asking, mainly, when they hear a change that is occurring, do they identify it correctly?

Pick the method that serves the aim of the research question. I don’t think there is a standard particularly but I like least the method of changing the denominator because it impacts the value of right answers or additional wrong answers - in other words, the points aren’t a linear function of right and wrong answers in your proposed method.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.