4
$\begingroup$

This question was provided in tutorial session:

Your friends Alice and Bob are both talented test-takers, but they can sometimes get distracted by their philosophies. Alice is an optimist, and she always wants to believe a statement is true. When given a true/false statement that is true, Alice gets the correct answer with probability 80%, but for false statements, she gets the correct answer only 35% of the time. On the other hand, Bob is a skeptic, and he tends to doubt the truth of statements. If a statement is true, he will get the correct answer with probability 15%, but if a statement is false, he will get the correct answer 85% of the time.

(a) Your teacher receives a bonus if all of her students achieve at least 50%, so she wants to design a true/false test on which both Alice and Bob will expect to receive at least this score. What percentage of statements on the test would you set to true in order to achieve this?

(b) Your teacher is now being bribed to make Alice's score as high as possible. However, she does not want Bob to fail the class. What percentage of statements should be true in order to maximize Alice's expected score while keeping Bob's expected score no less than 43%?

(c) You are now taking a true/false test with Alice and Bob. You cannot read the language in which the test is written, but you can see both Alice's and Bob's answers for each question; Alice's answers are independent of Bob's. You also know that each statement on the test is true with probability 50%. How should you fill out your answer sheet to achieve the highest possible expected score? What will your expected score be?

(d) Now imagine that you need to create an answer key for the test. You can look at the answer sheets for as many optimists (people who answer true questions correctly with probability 75% and false questions correctly with probability 40%) and pessimists (people who answer true questions correctly with probability 40% and false questions correctly with probability 90%) as you like. The students mistakes are independent of each other. You know that each statement on the test is true with probability 30%. How would you create an answer key that is at least 77% correct in expectation?

My approach::

Let C be the event that answer is correct and S be the event that the statement is true.

Then for Alice: P(C|S) = 0.8 and P(C|S') = 0.35

and for Bob: P(C|S) = 0.15 and P(C|S') = 0.85

(a)

We need to make P(C) >= 0.5 for both of them because both need to achieve at least 50%.

Suppose P(S) = x, P(S') = 1-x then for Alice:

P(C) = P(C|S)P(S) + P(C|S')P(S') = 0.8x + (1-x)*0.35 >=0.5

Solving we get x >= 0.33

Similarly for Bob we get x <= 0.43

So according to the question, % of questions set to true should be between 33-43%

(b)

Since Bob's score needs to be at least 43%, we have for Bob :

P(C) >= 0.43

P(C) = P(C|S)P(S) + P(C|S')P(S') = 0.15x + (1-x)*0.85 >=0.43

x<=0.7

So 70% statements should be true. I am not sure about this.

(c)

I have no clue about this one.

(d)

For Optimists: P(C|S) = 0.75 and P(C|S') = 0.4

and for pessimists: P(C|S) = 0.4 and P(C|S') = 0.9

P(S) = 0.3(given)

Optimists are expected to answer P(C) = 0.3*0.75+0.7*0.4=0.505 i.e 50.5% times correctly

and pessimists are expected to answer P(C) = 0.3*0.4+0.7*0.9=0.75 i.e 75% times correctly.

How can I achieve now at least 77% correct answer key because copying pessimists' answer only gives 75%?

Is my method correct? How to approach these kind of questions?

$\endgroup$

1 Answer 1

1
$\begingroup$

Your set-up is good, though maybe differentiate between $C_A$ (or just $A$) for 'Alice gets it correct' and $C_B$(or just $B$) for 'Bob gets it correct'. ALso, for convenience, I would use a letter that is more indicative than $S$ (which you chose because ... 'Statement is true'?) How about $T$ for 'Statement is True'.

a) You got the right equation, and for Alice you get $x \ge \frac{1}{3}$. But for Bob, if you think about it: if half of the statement are true and half are false, then Bob can be expected to get half of the questions correct. And if any more true statements are put on the test, then Bob will do worse. So, for Bob to get at least half correct, you need at least half of the statements false, which is the same as having at most half of the statements being true. So, assuming you had the percentage for Alice correct, you need to have between $\frac{1}{3}$ and $\frac{1}{2}$ true statements.

For b), you should point out that the more true statements are on the test, the better Alice does, and the worse Bob does. So, to maximize Alice's score, it is indeed a matter of minimizing Bob's score to $43$%. So, just set the equation to $0.43$. That is, instead of saying that we want:

$$0.15x + (1-x) \cdot 0.85 \ge 0.43$$

you can say that we want:

$$0.15x + (1-x) \cdot 0.85 = 0.43$$

For c) note that the probability of Bob getting the right answer on any question is $0.5*0.15+0.5*0.85=0.5$ (i.e Bob can be expected to get half right), while the probability of Alice getting the right answer on any question is $0.5*0.80+0.5*0.35=0.575$. Put differently, you won't learn anything from Bob's answers, but you can learn something from Alice's answers. In fact, you can use Bayes' Law to see that when Bob answers 'True', he is correct $50$% of the time, and same for when he answers 'False', so again, there is no information to be gained from Bob's answers. But when Alice answers 'True', she is more likely correct than incorrect, and same with her answering 'False'. So, the best strategy is to simply copy Alice's answers, with which you can expect to get $57.5$% right. (This question could have been soooo much more interesting if Bob's answers were revealing, and if you knew the number of true questions was not exactly half!)

ADDENDUM

Aha! So d) is exactly that 'much more interesting question' I referred to at the end of c). Yes, it seems paradoxical to get $77$% accuracy when the best test-takers (the pessimists) can only score $75$% accuracy, but there are at least 3 things going on which makes this different from c)

First of all, the number of True statements is not the same as the number of False statements. So, this creates an asymmetry in the first place. Of course, it creates an asymmetry in favor of the pessimists, and as your calculations show, they will indeed do much better on the test than the optimists, so can we really do better than the optimists? Well:

Second, note that the optimists score (just) above 50% ... so you can learn something from the optimists. In fact, while you should most likely answer 'True' when everyone answers 'True', and same for 'False', it is conceivable that the best thing to do when the optimists say 'True' and the pessimists say 'False', is to go with the pessimists and say 'False', while when the optimists say 'False' and the pessimists say 'True', the best thing might be to go with the pessimists. In other words, there is of course no need to go with the answers of one group, and especially where there is a disagreement of one kind, to trust the one group better than the other, but with a disagreement of a different kind, this gets reversed.

Now, given how poor the optimists are doing on the test, it turns out (I did the calculations) that when all optimists say one thing, and all pessimists say another, it is still always best to go with the pessimist's answer, so you'd still end up with the pessimists' score of $75$%, but there is one more crucial difference:

Third, you now have multiple optimists and pessimists. Now actually this raises a practical question as far as your strategy goes: what do you do when not all optimists agree? Or not all pessimists? Well, one strategy is to figure out the majority answer of the optimists, as well as the majority answer of the pessimists. But given that the pessimists are so much better on this test, you'd probably still just go with the pessimists' majority answer.

OK, but what would you say when a huge percentage of the optimists say 'False', but only a small majority of pessimists say 'True' to some question? Well, the optimists are inclined to say 'True' to most questions, so if a vast majority (if not all) of them say 'False', that should really give you some pause: maybe they all see something in the question that makes them break their 'habit'. Of course, we also have a majority of pessimists saying 'True', which should give you pause as well ... but if that is just a slight majority, then that would indicate they are much less confident. And so in a case like that, it may be better to go with the optimists' majority answer, rather than with the pessimist's majority answer.

Long story short: you can learn something from both the optimists and the pessimists, and you can especially learn something from the proportion in which they answer the question.

Now, how do you translate this into an actual strategy? One strategy that takes all of this into account is to simply go with the majority opinion for every answer, not considering as to whether the answer was given by an optimist or a pessimist. (in fact, it is not clear from the question whether you even know who the optimists are and who the pessimists).

But if you do know who the optimists and the pessimists are, the strategy would involve looking at the proportion of the groups answering a certain way, and then asking a kind of Bayesian question: "given that $X$% of the optimists answered this way, and $Y$% of the pessimists that way, what is the probability of the correct answer being such-and-so?"

Frankly, I am not exactly sure how to do that in general ... but maybe you could consider looking at just 2 optimists and 2 pessimists ... so you could use the Bayesian theorem to compute things like "given that both optimists said 'True', and given that 1 pessimist said 'True' but the other 'False', what is the chance the Statement is True?"

That will be a lot of work ... you'd need to consider 9 possible outcomes (in each group: both say 'True', both say 'False', or a split) ... but it's mostly just tedious work ... and maybe that will get you to a $77$% 'accuracy ...

Now, just to demonstrate how to do this kind of calculation, I will analyze the case most similar to what I just described, which is where both optimists say 'False', and where one pessimist says 'True' and the other 'False'.

OK, so first I will insist on using a $T$ for 'statement is true' and a $F$ for 'statement is false' (rather than your $S$, which by itself does not easily reveal what it stands for). Also, let's use $T_O$ for 'optimist answers true', $T_P$ for 'pessimist answers true', $F_O$ for 'optimist answers false', and $F_P$ for 'pessimist answers False'.

So, we are given:

$P(T) = 0.30$

$P(F) = 1-0.70=0.30$

$P(T_O|T) = 0.75$

$P(F_O|T) = 1-0.75 = 0.25$

$P(F_O|F) = 0.40$

$P(T_O|F) = 1-0.40 = 0.60$

$P(T_P|T) = 0.40$

$P(F_P|T) = 1-0.40 = 0.60$

$P(F_P|F) = 0.90$

$P(T_P|F) = 1-0.90=0.10$

Let's compute some further useful probabilities. For example, what is the chance of both the optimists saying 'False' and the pessimists be split (let's call this event $E$), assuming the statement is true ? By independence of the answers, that would be:

$P(E|T) = 0.25^2 \cdot 0.4 \cdot 0.6 \cdot 2 = 0.03$ (the $2$ is because the split among the pessimists can happen in two ways)

Likewise:

$P(E|F) = 0.4^2 \cdot 0.1 \cdot 0.9 \cdot 2 = 0.0288$

This means that the probability that event $E$ happens is:

$P(E) = P(E|T) \cdot P(T) + P(E|F) \cdot P(F) = 0.03 \cdot 0.3 + 0.0288 \cdot 0.7 = 0.009 + 0.02016 = 0.02916$

By the Bayesian Theorem, we can then calculate the chances of the statement being true in the case of event $E$:

$P(T|E) = \frac{P(E|T) \cdot P(T)}{P(E)} = \frac{0.03 \cdot 0.3}{0.02916} \approx 0.309$

And so, this tells you to go with the optimists answer of 'False' (as you will have a chance of $P(F|E) = 1-P(T|E) = 1-0.309=0.691=69.1$% of getting the correct answer), even though half of the pessimists say 'True'.

So yes, do these calculations for all 9 events, figure out what the best strategy is in each case, and then calculate your expected accuracy. Good luck! (and let me know what happens ... maybe looking at 2 people from each group is not enough ... 3 would indeed be interesting, because then you would always have a majority answer for each group ... and again the interesting case would be where all optimists say 'False', and where two pessimists say 'True' and one says 'False' ... for then the 'majority' answer would be different between the two groups, but I think (eye-balling this) in that case it would be best to go with the optimists' majority answer of 'False')

ADDENDUM 2

OK, I couldn't bear the suspense ... if you follow the strategy of looking at the answers of two optimists and two pessimists, and pick what was most likely the correct truth-value of the statement given their answers, you get to ... drumroll ...) $77.6$%! So yes, that's it!

$\endgroup$
11
  • $\begingroup$ I don't understand by what you meant by "set the equation to 0.43" $\endgroup$
    – chelsea
    Commented Aug 19, 2017 at 20:09
  • $\begingroup$ @ Chelsea I meant instead of saying $\le 0.43$ make it $=0.43$. I'll update my answer ... $\endgroup$
    – Bram28
    Commented Aug 19, 2017 at 20:11
  • $\begingroup$ I have added part d that I had missed yesterday. I tried to solve but stuck at something. can you please review my approach? $\endgroup$
    – chelsea
    Commented Aug 20, 2017 at 14:28
  • $\begingroup$ Thanks..very nicely explained :) $\endgroup$
    – chelsea
    Commented Aug 27, 2017 at 14:35
  • $\begingroup$ @chelsea You're welcome! Part d) was rather fun ... were you able to do all those calculations yourself? I recommend using a spreadsheet. $\endgroup$
    – Bram28
    Commented Aug 27, 2017 at 15:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .