0
$\begingroup$

I have around 50K multiple-choice questions and each has been answered by three different students. Each question has 5 options. There is a high chance that a student answered more than one question (1-300 questions). We do NOT know what the correct answers are and we want to guess them based on the behaviour of the respondents. I was wondering if there is any field in the math that can handle questions like this. I hope here is the right place to raise this question.

$\endgroup$
2
  • $\begingroup$ Three students each answered 50,000 questions? Your Institutional Review Board allowed this? $\endgroup$ Aug 11, 2020 at 3:57
  • $\begingroup$ No, there are more than three respondents in this problem (something around 500 respondents). This is a crowdsourcing project, there is no restriction on the number of questions per respondent. $\endgroup$
    – Sarah
    Aug 11, 2020 at 5:06

1 Answer 1

0
$\begingroup$

I really enjoy your question.

Since we want to know the answers we kind of need to know who we can or can not trust. Let $s_1, s_2,\cdots, s_n$ be the students ($n\sim 500$). Now let $t_i$ be how much we trust student $s_i$, if $t_i<0$, $s_i$ is a bad student, if $t_i= 0$, $s_i$ is ordinary and if $t_i>0$, $s_i$ is a good student . If we can trust in $s_i$ and $s_j$ gives the same answer as $s_i$ then we can trust $s_j$ a bit more. I'll assume that $t_i$ is the mean between all agreements (with positive weights) and disagreements (with negative weights) between $s_i$ and the others, then $$ t_i = \frac{1}{|Q_i|}\sum_{q\in Q_i}\frac{1}{2}\left( \sum_{ j: s_i\text{ agree with }s_j\text{ in }q }t_j - \sum_{ j: s_i\text{ disagree with }s_j\text{ in }q }t_j \right) $$ where $Q_i$ is the set of all questions $s_i$ answered. We can also assume that, overall, we can trust people's answers, so $$ t_1+t_2+\cdots + t_n = 1 > 0 $$

That gives you a linear system. The system has $n$ variables and $n+1$ equations, I don't know if it is possible that this system has no solution or has multiple solutions. It need to be studied. But you can code and see what happens.

If the system has a solution we can rank the answers by how reliable the students are. That also gives us an indicator of how much we can trust the solution.

That was my approach. But a lot of mature things exist in the literature.

You can also see this problem as a "propagation of trustiness" were the more reliable are the individuals that agree with you, the more reliable you are. That is exactly what Google Page rank originally did, the students were webpages and links between webpages were answers in commom. It can be solved with a bit of linear algebra and Markov Chains, see this or this or even this (with the Python code, but in Portuguese). Other ways of propagating trustiness and distrustiness can be viewed here.

After solving the trustiness problem, you will always need to guess if the more trusted people were the bad or the good students, or just good representants of the commom sense. If they are students answering about math, history, etc., like SAT questions, then, I think, it makes sense to assume that the most trusted are the best.

$\endgroup$
2
  • $\begingroup$ Thank you so much for the detailed answer. I found this article "Maximum likelihood estimation of observer error-rates using the EM algorithm" as something that I was looking for. It makes a. confusion matrix for each respondent to measure their error rates and iteratively estimates the correct answer until convergence. $\endgroup$
    – Sarah
    Aug 12, 2020 at 14:23
  • 1
    $\begingroup$ In the end it is all about finding the final distribution of a Markov chain :) $\endgroup$ Aug 12, 2020 at 14:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .