2
$\begingroup$

I want to generate $n-1$ wrong answers $a_2,\ldots,a_n$ for a multiple choice test given a correct number $a$. The problem is that I don't want the user to be able to guess the correct option with an accuracy of more than $\frac{1}{n}$ if the user doesn't know the correct number. Is that possible and if yes, how?

Ideas

Always draw from the same interval

Ignoring the correct value $a$, just always choose $a_i$ randomly from $[$lower bound, upper bound$]$, e.g. [0,10000]. The problem is that if I ask for e.g. the year of the french revolution ($a=1789$), the user knows that it cannot be e.g. in year 2345. Even if I set the upper bound to the current year, giving the years 110, 720 and 2001 as wrong options will be too easy. Even if the answer isn't a year, a smaller value will be more probable in general.

Draw numbers a bit smaller, larger or around the correct number

If I generate slightly smaller numbers (lets say from the interval $[0.9 \cdot a,a)$) than the correct answer, the user just always chooses the largest one. If I always generate slightly larger ones, $(a,1.1\cdot a]$, the smallest one is always correct. And if I generate numbers around the number, $[0.9 \cdot a,1.1 \cdot a]$, the one in the middle will be more often correct than the others.

Two Step Process

I could first choose a method randomly, 1/3rd of the time I choose smaller, 1/3rd of the time larger and 1/3rd of the time "around" the number. Now I can still break this because if I see that $\frac{\max(a_i)}{min(a_i)} > 1.1$, then I know the last option was chosen and I can choose the middle one. If I change the "around" interval to $[0.95\cdot a,1.05\cdot a]$, the quotient could still be slightly over 1.1. I could use $[0.95\cdot a,0.95\cdot1.1 = 1.045 \cdot a]$ but now $a$ isn't in the center of the interval any more which can be exploited with a uniform draw of random numbers.

Is there any way to create random numbers similar to the correct numbers that don't give any information about it? Maybe I need to use a random number generator that doesn't have a uniform probability in the interval? Or should this exploitability just be factored into the scoring?

P.S.: I guess $0.9\cdot a$ should be $\frac{a}{1.1}$ so that the quotient is the same.

$\endgroup$
9
  • 2
    $\begingroup$ I think this is potentially interesting but not at all well specified. You suggest that the test taker has some information about the correct answer ($2001$ can't be right for the French revolution.) Until you specify that kind of knowledge you can't quantify how much information your wrong answers convey. What precisely might "similar to the correct number" mean? In multiple choice questions wrong choices are often carefully chosen to be the results of wrong methods. $\endgroup$ Apr 11, 2019 at 12:18
  • 1
    $\begingroup$ If the question are for some fixed real life facts, you need to pick a suitable range/class of possible answers. This has to depend on the particular fact and also has a huge influence on how hard the questions will be. For the French revolution example: if the wrong answer lie in 1 to 2000 it will be an easy question, if all the wrong answers are between 1750 und 1800 it will be a lot harder. That is a choice for the question poser, there is no maths solution to that. $\endgroup$
    – quarague
    Apr 11, 2019 at 13:14
  • 2
    $\begingroup$ This question should go to an educators' site, not a math site. They will be familiar with the copious literature on the topic. $\endgroup$ Apr 11, 2019 at 13:34
  • 1
    $\begingroup$ I repeat: there is a copious literature on the subject of constructing multiple choice questions, and the people familiar with that literature should be at the educators' sites. $\endgroup$ Apr 11, 2019 at 13:44
  • 1
    $\begingroup$ matheducators.stackexchange.com/questions/13058/… $\endgroup$
    – user7610
    Apr 12, 2019 at 15:20

2 Answers 2

1
$\begingroup$

Here are some thoughts about an abstract version of your question.

Assume a finite set of $A$ of possible answers (your questoin suggests that will be an interval of integers). Randomly choose $n-1$ elements from the set of wrong answers. Then provide as choices the correct answer and your wrong answers (place the correct answer at random).

If the test taker knows the answer they will choose it. If they have no information they will guess and succeed with probability $1/n$. If they have some information about the correct answer then they will be able to increase their probability of guessing correctly to some value greater than $1/n$. You can't avoid that unless you know more about how they are thinking.

If you knew their personal probability distribution over the set $A$ of possible answers (something between uniform and certainty) I think you could design a random set of wrong answers that provided minimal information in some sense. Quantifying that strikes me as an interesting and probably difficult problem. When they are certain and correct nothing you do will fool them. If they are certain and incorrect in a way you know should you allow their wrong choice to be one you include along with the correct answer? What would your best strategy be if they thought two of the possible answers were equally likely to be correct?

I suspect this is the kind of game cryptanalysts play,

$\endgroup$
1
  • $\begingroup$ I cannot avoid them having a higher chance of choosing correctly if they have some information but I would like minimize that chance. For example if they know the century of the French revolution, and the wrong answers lie in other centuries, the player will always guess correctly, which seems a bit high for me. I don't have information about the mind of the players, but I will try to make the numbers as close as possible to the correct one, I assume this will minimize the information given. $\endgroup$ Apr 15, 2019 at 10:06
1
$\begingroup$

Let me try to think of this as a classification problem. The student has a set of options and needs to decide whether the option is a correct answer or not. A probabilistic view is also possible, where the student gives odds on each of the answers.

Each student has different abilities to perform the classification. On one end of the spectrum, student knows the correct answer even before looking at the options. On the other end, student thinks all answers are equally likely, because the student knows nothing to help choose.

Considering the French Revolution question, I can imagine that students may progress on the following ability ladder between the two extremes.

  1. I know it happened some time in the past
  2. There was this bare chested painting of woman on barricade, and lots of beheadings later. No clue when, though.
  3. I know it had something to do with the US Independence war, but I don't know which was first
  4. US Independence war was first
  5. I faintly remember a collection of dates which I memorized for the test, but I don't know which is which.

This will be different for each student, though, so different options are differently likely for each.

In education, there is item response theory, which tries to estimate a model of student's ability by presenting large set of students with large set of questions, and mutually calibrating the questions on students and students on questions. Various models of various complexity can be used to quantify student abilities.

I believe that the question cannot be answered without actual experimentation of some kind (or actual teachers experience substituting for it). If I had the data, I don't know for certain what I'd do with it, but I believe something can be done then.

$\endgroup$
3
  • $\begingroup$ Unfortunately I don't have data about the players' abilities and cannot experiment with them as I generate the questions automatically beforehand, but in that case it seems that there is no general solution. However I would like to view the case more abstractly, without players, as a statistical problem. $\endgroup$ Apr 15, 2019 at 10:09
  • $\begingroup$ I think you cannot do this without experimentation. You may be able to use experiments done with one set of questions to another set of questions, or substitute common sense and experience, but there has to be some reality check of this sort. If only to evaluate the properties of the choice generation method, to choose the best one. $\endgroup$
    – user7610
    Apr 15, 2019 at 12:45
  • $\begingroup$ "I don't have data about the players' abilities and cannot experiment with them as I generate the questions automatically beforehand". The strength of IRT is that you do not need to start with data about players abilities nor about question difficulty. You can estimate them both in tandem from studying how players interact with the questions. $\endgroup$
    – user7610
    Apr 15, 2019 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.