I am interested in analyzing user performance at a per user level. Users have the opportunity to engage with an app by answering trivia questions and can choose how many questions they want to answer. Currently performance is simply computed by the following expression:(# of questions they answered correctly)/(# of questions attempted). I then rank users based off this performance.

However, this approach is a bit naive. If a user answers 1/1 questions correctly and another answers 20/20 correctly, they still have the same "performance". Or even if a user answers 1/1 correctly and another answers 14/15 correctly, the 1/1 has a higher performance ranking even though we don't have very much information to generalize performance.

Additional motivation could come from the following example: if I want to track how user performance changes on a day-to-day basis. Let's say user A answers 1/1 correctly yesterday and 9/10 today. Using my current approach the difference would be (9/10) - (1/1) = -0.1. Technically the performance dropped but I feel better about the user answering 9/10 than 1/1.

I would love suggestions and input for incorporating some sort of "confidence" in the computed performance (not sure how this would work). While there is nothing wrong with my current approach, I feel uncomfortable that people with such low activity can be considered high performing so it could be useful to somehow incorporate the number of attempted questions. I don't really care between distinguishing 40/50 and 160/200 (since 50 seems like a pretty good size generalize performance).

  • $\begingroup$ Assuming binomial distributions, your confidence interval is just $\pm 1/\sqrt{n}$ where $n$ is the number of questions attempted. Use simple students $t$-test to determine if one accuracy is statistically significantly higher or lower than another. $\endgroup$ – David G. Stork Jan 16 at 22:03
  • $\begingroup$ You might get better answers by posting to the stack exchange statistics site: stats.stackexchange.com $\endgroup$ – awkward Jan 17 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.