I am interested in analyzing user performance at a per user level. Users have the opportunity to engage with an app by answering trivia questions and can choose how many questions they want to answer. Currently performance is simply computed by the following expression:
(# of questions they answered correctly)/(# of questions attempted). I then rank users based off this performance.
However, this approach is a bit naive. If a user answers 1/1 questions correctly and another answers 20/20 correctly, they still have the same "performance". Or even if a user answers 1/1 correctly and another answers 14/15 correctly, the 1/1 has a higher performance ranking even though we don't have very much information to generalize performance.
Additional motivation could come from the following example: if I want to track how user performance changes on a day-to-day basis. Let's say user A answers 1/1 correctly yesterday and 9/10 today. Using my current approach the difference would be (9/10) - (1/1) = -0.1. Technically the performance dropped but I feel better about the user answering 9/10 than 1/1.
I would love suggestions and input for incorporating some sort of "confidence" in the computed performance (not sure how this would work). While there is nothing wrong with my current approach, I feel uncomfortable that people with such low activity can be considered high performing so it could be useful to somehow incorporate the
number of attempted questions. I don't really care between distinguishing 40/50 and 160/200 (since 50 seems like a pretty good size generalize performance).