Suppose we have a Multiple Choice Test with the following characteristics:
- There are M available multiple choice Questions on the system
- There are N users participating
- Each user replies to K questions, for convenience let's assume K < M
Eventually, we can define the difficulty level of each question by computing the population's probability of answering correctly eg
P(correct) = CorrectAnswers/TotalAnswers
I need to create an alternative to ELO Rating System for each player. I could just use the Total P(correct) for each user, but this is not indicative of user's skills, because each question set is random and some users may be given easier questions than others.
Till now, I just grouped the questions based on their difficulty P(correct) and used each 10% percentile to create 10 different sets. So, I now have 10 ranking lists that somehow estimate users skills on each of the 10 question levels. Users with better accuracy in each list are ranked higher.
But, I need a generic single Ranking List algorithm; I wonder if there is a general formula that applies to the whole set of questions, but on the same time it uses the difficulty level P(correct).
FYI: the problem has some similarities to creating a Ranking List on basketball players by merging accuracy of both 2-points and 3-points shots. Is there a joint Ranking formula for that?