0
$\begingroup$

I've been wondering whether it's possible to improve the chances of getting fewer questions wrong on multiple choice exams using probability. I thought of this method, but I'm not sure if it helps.

Suppose I'm writing a multiple choice exam. I'm fairly confident I know the material, and have checked my answers over one or two times. Say there is not enough time to check over everything carefully another time.

There are a couple sections on the exam where I get "hotspots" like AAAA or CCCCC (4 or more repeating letters). Let's call the list of answers for a section a pattern. Should I give more weight to checking these questions?

I know these are true (I think these are true):

  • It is more likely that the correct pattern is not several repeating letters, rather than it is
  • It is not more likely that any other particular pattern is correct, rather than a particular pattern of repeating letters

In this case, I am equally confident in all my answers, so I'm only looking for careless mistakes, or thinking more carefully over some difficult questions. The multiple choice exam is 4 or 5 choices (and how does the number of choices affect the probability?)


Consider another, similar(?) case: there is a question I am completely 50/50 on A or B. If I choose, A, it creates a pattern of 5 recurring letters (AAAAA). Should I go with A or B, or does it matter?

The obvious answer is that it is still 50/50. However, I feel like there may be something more about it, like with the Monty Hall problem (it's not about A or B, it's about A or not A).


Edit: I guess it wasn't clear. I'll try to put this in more mathematical terms, although I'm worried I'll skew the question when rewording it, especially because I'm not primarily a math person.

  • There is an ordered set of letters (the answer key), which is random
  • I am trying to figure out the original ordered set of letters. I should be mostly right, but there may be some mismatches
  • Is it more likely that a section with repeating letters contains a mismatch rather than in another section of the set?

It seems like the answer is no... but can someone explain it more clearly? There seem to be two conflicting factors:

  • It is more likely that a subset of consecutive elements are not the same, rather than that they are
  • Any two possibilities are equally likely
$\endgroup$
  • 2
    $\begingroup$ This depends more upon the author of the test than anything else. Is this the kind of professor who would avoid giving a test where every answer is C? Or is this a professor who would give no regard to patterns of any sort in the answers? "Now, a clever man would put the poison into his own goblet, because he would know that only a great fool would reach for what he was given. I am not a great fool, so I can clearly not choose the wine in front of you. But you must have known I was not a great fool. You would have counted on it, so I can clearly not choose the wine in front of me." $\endgroup$ – nispio Nov 6 '13 at 5:43
  • $\begingroup$ @nispio assume the professor isn't overthinking it; it's just random, but not intentionally-completely random, if that makes any sense $\endgroup$ – Raekye Nov 6 '13 at 6:44
  • $\begingroup$ @nispio s answer is well though out. But if I had a question I was completely uncertain on, and could break a pattern with one of the choices, that is the way I would bet. No guarantees, of course, but it is my read on human nature. $\endgroup$ – Ross Millikan Nov 16 '13 at 3:45
4
$\begingroup$

If the test was made without any regard to distributions/patterns, then probability could not help you because each question would be independent and uniformly distributed. This means that nothing can be inferred about one answer based on information about the other answers, and that no answer is more likely than the others. However, I would argue that this is a less likely scenario becuase humans are notoriously bad random number generators. Making the answers be truly i.i.d. would usually require the use of a PRNG or similar means. This seems much more likely in the case of standardized tests, and somewhat less likely when you receive simple "circle the correct answer"-style tests (as opposed to OMR-style answer sheets)

So let's assume that the answers are either not independent, or that they are not uniformly distributed. Now can you use probability to help you? Let's look at each case:

Uniformity

  • The professor may favor some answers more than others
  • If the distribution of answers is not uniform, then for any given problem there are answers that are more likely to be correct than others. Notice that this has nothing to do with patterns, or independence. You could easily generate a test where the answers are independent but not uniformly distributed. For example, roll a six-sided dice, and assign the answers as follows: 1=A 2=B 3=D 4=C 5=C 6=C. You will notice that C is the most likely answer for any given question but the dice rolls are still completely independent.

    Can you use this information to help you take the test? Maybe. If you knew before ever seeing the test that this particular professor favors C as an answer, and that previous tests from this professor had C as the correct answer more than 50% of the time, you could potentially leverage this information. But how? If I know that any given question has a probability of 0.5 of having C as the correct answer, but I observe that in my completed test only 40% of the answers are C, the best I can do is to re-evaluate all of the non-C answers, and see if I think that C might be a better answer. This will not be very fruitful, especially because the fact C is more likely does not mean that if you have more Bs than Cs that you have done anything wrong. C can be more likely to occur than B and yet B can show up more frequently on a given test.

    Instead, the best way to use this information is on a per-problem basis. Read the problem and the answer choices. Throw out any answers that are obviously not correct, and then evaluate the ones that might be correct. From the remaining answers you can combine the known distribution with your knowledge about the problem to create a likelihood for each answer. The answer with the highest likelihood wins.

    Independence

  • The professor may avoid long sequences of the same answer
  • In this case, you can use a conditional probability that a B answer will occur given that the previous n answers were B. You can again combine this information with your knowledge about the problem, and then maximize the likelihood. The reason that this won't work is that you would need very reliable information about how likely the professor is to allow 2 Bs in a row, 3 Bs, 4 Bs, etc. This information could possibly be extracted if you had a large enough sample size of prior tests from this same professor, but without getting inside the professors brain, it would still require a lot of inference on your part. It is entirely possible that this professor has no problem with allowing 6 Bs in a row, but since such a sequence only has a 0.02% chance of occurring naturally, you may collect 1000 prior tests and still never see this pattern.

  • The professor may avoid seemingly "non-random" patterns
  • The arguments against this one are at least as strong as the previous. Without specific knowledge of what the professor considers to be a "pattern", and to what degree he is willing to allow patterns to occur naturally, it would be nearly impossible to extract this information from prior tests.

  • The professor may attempt to create a "perfect" uniform distribution
  • In the extreme case, you might know that there were exactly the same number of each answer on the test. If you ended up with two more Bs than Cs you could assume that one of the Bs was meant to be a C, and attempt to locate the offender.

    Having said that, this won't work with anything short of a "perfect" distribution in which each answer is represented exactly the same number of times. If the professor just has a soft goal of trying to distribute the answers evenly, then you really can infer almost nothing.

    Conclusion

    There is very little to be gained from "probabilistic test taking" unless you have access to information about how the test is generated. Unless the professor has explicitly stated the rules he uses to distribute the answers, the only way to obtain this information is through statistical inference after processing "sufficient" historical data. As described above, this may not be just impractical, but impossible. The professor may not have written enough tests in his lifetime to make reasonable inferences about how his tests are generated.

    But hey, when in doubt, mark C. That's what I do.

    $\endgroup$
    1
    $\begingroup$

    No, there is none. We could assume that the exam-maker will avoid using patterns such as AAAAA (when preparing the exams). But, then we could assume that the exam-maker will actually put patterns such as AAAAA as the smart students thought that the exam-maker wouldn't and so on. Better study hard instead of worrying about this stuff. Also, it is likely that the answers to serious multiple choice exams are permuted by the computers, so none of the psychological methods will work there. In any case, perhaps the question should be reformulated to look more like a mathematical question.

    $\endgroup$
    • $\begingroup$ Yeah I am interested in a mathematical explanation (of why which one is correct), and by the way, I already know the material ;) That's why I'm done checking over 2-3 times, and wondering what's the best way to eliminate any careless mistakes, or make a better choice on the few I'm less confident on. So to be clear: I'm don't care about the psychological factors. I'm asking if there's a better choice by probability. $\endgroup$ – Raekye Nov 6 '13 at 6:48
    • $\begingroup$ There isn't a better choice by probability, unless you think the test writer deliberately skewed things. But how could you know? $\endgroup$ – Betty Mock Nov 6 '13 at 7:21
    1
    $\begingroup$

    The idea that 'AAAAA' is less likely than 'ABAAAB' has a tacit assumption that somehow probabilities of individual options are affected by other options. As Lord Soth pointed out, this would then mean that exam-maker is intentionally avoiding repeating patters but this may very well not be the case.

    Unless you know the exam-maker and can make inferences about how he/she/it creates exams and resulting question/answer patterns, the probability for both repating patterns and non-repeating patterns is the same ie. each and every option is independent. Maybe it helps to think this as a simple case of probability where you toss a four-sided dice having sides from 1 to 4: the dice is equally likely to produce '1111' and '1234' and every other combination because every time you toss the dice it has 1/4 probability of producing each possible outcome ie. number 1,2,3 and 4 (I'm going to leave out the possibility of the dice landing on it's side and not producing a unambiguous result).

    $\endgroup$

    Your Answer

    By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

    Not the answer you're looking for? Browse other questions tagged or ask your own question.