18
$\begingroup$

You put a cat into a toxic box, it might be dead or alive after an hour. Two witches $A$ and $B$ have the ability to predict the status of the cat with the accuracy of $p_1$ and $p_2$, respectively.Suppose their prediction are independent. Both of them claim that they "see" the cat is still alive in the future. Question: from their words, can you calculate the probability that the cat is still alive after an hour?

I have two solutions, I am so confused about them, especially the second one, please have patience and feel my confusion.

Solution 1

Two possibilities:

  1. $A$ right, $B$ right, so cat is alive
  2. $A$ wrong, $B$ wrong, so cat is dead

One right and one wrong is impossible, therefore the probability that the cat is alive $p_1p_2/(p_1p_2+(1-p_1)(1-p_2))$

Solution 2

I will translate A has the accuracy of $p_1$ to $$ p_1 = P(L)P(A_L|L)+P(D)P(A_D|D) $$ above formula says if the cat is alive, $A$ predict it is alive; if the cat is dead, $A$ predict it is dead.

Similarly: $$ p_2 = P(L)P(B_L|L)+P(D)P(B_D|D) $$ Because the cat is either alive or dead, so $P(L)+P(D)=1$

We want to calculate the probability that the cat is alive based on the words of $A$ and $B$:

$$ P(L|A_L,B_L) = \frac{P(A_L,B_L|L)P(L)}{P(A_L,B_L)} = \frac{P(A_L,B_L|L)P(L)}{P(A_L,B_L|L)P(L)+P(A_L,B_L|D)P(D)} $$

which seems not the same as solution 1 as far as I can tell.

This solution makes me very uncomfortable about the meaning of $P(D)$ and $P(L)$, are they the unknown probability of the cat is dead or alive if I have no witches words?

Edit

I found the solution 1 is the result of explaining the accuracy of prediction differently than solution 2. Explicitly, it explain $p_1=P(D|A_D)=P(L|A_L)$, which means, when $A$ says the cat is dead, the probability that the cat turn out to be dead is $p_1$; when $A$ says the cat is alive, the probability that the cat turn out to be alive is also $p_1$.

Therefore,

$$ P(L|A_L,B_L)=\frac{P(L|A_L,B_L)}{P(L|A_L,B_L)+ P(D|A_L,B_L)}= \frac{p_1p_2}{p_1p_2+(1-p_1)(1-p_2)} $$

Edit2

As discussed, I think the most confusing part is how to properly define the accuracy of the witches, do you think there are ambiguities in the definition of this quantity? At least three candidates make some sense to me:

  1. $p_1=P(A_L|L)=P(A_D|D)$

This says: if the cat is alive, then the witch predict it as alive with probability $p_1$; if the cat is dead, then witch predict it as dead with probability $p_1$;

  1. $p_1 = P(L|A_L) = P(D|A_D)$

This says: if the witch says the cat is alive, then the probability that the cat turn out to be alive with probability $p_1$; same as when the witch says the cat is dead.

  1. $p_1 = P(A_L|L)P(L) + P(A_D|D)P(D)$

This says: the cat is either dead or alive, if it is alive, the witch predict it as alive; if it is dead, the witch predict it as dead; the sum of them should be the accuracy of the witch. As pointed out by one answer, this definition allows that $P(A_L|L)\neq P(A_D|D)$, but is this a big problem?

Which of the above three make most sense to you, which is nonsense and which make some sense?

$\endgroup$
26
  • $\begingroup$ Well, you need to know something about the independence of the predictions. If witch $2$ just repeats whatever witch $1$ says, then there is no new information in her prediction. $\endgroup$ – lulu Dec 12 '16 at 15:20
  • $\begingroup$ What a strange question. But if the predictions are independent, this looks reasonable at first glance. $\endgroup$ – The Count Dec 12 '16 at 15:20
  • 2
    $\begingroup$ I would add that you also need to have a prior for the cat's life expectancy. Suppose that cat dies with probability $1$....both witches just toss weighted coins to make their predictions. Then their predictions give no information at all about the poor feline. $\endgroup$ – lulu Dec 12 '16 at 15:23
  • 2
    $\begingroup$ I don't, sorry. The error comes when you re-interpret the definition of $p_i$. As defined, $p_i$ is simply the probability that witch $i$ says the truth. That is not the same as saying that whatever she says is true with probability $p_i$. I speak the truth with very high probability but that doesn't mean that if I say "$1+1=3$" then $1+1$ is probably $3$. $\endgroup$ – lulu Dec 12 '16 at 16:23
  • 1
    $\begingroup$ You need a prior AND you need to know how accurate each prediction is. I don't think this is a tricky question...to me the answer to the question is simply "No. There is not enough information to answer." $\endgroup$ – lulu Dec 12 '16 at 16:48
4
+25
$\begingroup$

Good question. I especially liked your journey (seen through your edits) as you were grappling with this problem. I wish more questions on this site were written like this.

The very short answer is: Yes, we need to disambiguate the word "accuracy" and we also need the prior probability.

Let's start with your latest question (in edit 2) because this is indeed a crucial question.

What does prediction accuracy mean?

You are right to wonder. This is not a well-defined term. When we are making predictions about a binary event (for example: cat dead or alive, or as another example: a person either has the disease or not) and we want to measure how good a prediction method is, we should use two numbers: The rate of true positives and the rate of true negatives. These are usually called sensitivity and specificity respectively. Using your notation and considering the prediction of witch $A$, these are the quantities: $P(A_L|L)$ and $P(A_D|D)$. So sensitivity (true positive rate) is the probability that witch $A$ predicts the cat alive, given that the cat is alive (or will be alive). Specificity (true negative rate) is the probability that witch $A$ predicts the cat dead, given that the cat is (or will be) dead.

You have already identified these quantities are important, and you are wondering how accuracy relates to them. The most straightforward interpretation is your first one, namely that both sensitivity and specificity are equal to the number given as "accuracy". You can view this interpretation as the prediction method being equally "accurate" both for the positive events (cat alive) and negative events (cat dead). Even though this is probably the most straightforward interpretation, the problem description should be explicit about what it means by the single quantity "accuracy".

So the takeaway message is: prediction quality is defined by two numbers (which often people presume equal and give you just one number).

Your second interpretation

Let's look at your alternate interpretations. Could we define accuracy as $P(L|A_L)$? We are getting a bit into philosophical territory now. Sure, we could define whatever we want, and we could name $P(L|A_L)$ as some kind of accuracy metric but usually this is the quantity that we are looking for (i.e., given some predictions what is the probability of the actual event). One can argue that there is no harm in having our definitions serve us the answers on a plate, but there is an issue with this definition. $P(L|A_L)$ depends on $P(L)$, so it cannot be seen as a pure reflection/metric of the prediction method. $P(L)$ is the prior probability (i.e., general prior knowledge of the probability of the event happening irrespective of predictions). As you have found in your solution 2 (employing Bayes rule), and you discussed in the comments, you need to know $P(L)$. But it is better not to mix a sense of method accuracy and the prior in one quantity. The accepted norm of using sensitivity and specificity avoids exactly this.

More importantly, with our specific problem this definition becomes really problematic, because we have two (presumably different) "accuracies" $p_1$ and $p_2$. If we define $p_1 = P(L|A_L)$ and $p_2 = P(L|B_L)$, then what is $p= P(L|A_L,B_L)$? There is a valid solution only if $p=p_1=p_2$, otherwise the problem description contradicts itself.

Your third interpretation

How about your third interpretation? Accuracy $= P(A_L|L)\cdot P(L) +P(A_D|D)\cdot P(D)$. When I first saw this, I went "Huh?", but then I understood the intention. It is still a rate/probability (quantity between $0$ and $1$) and it represents an aggregated truth rate weighted with the prior probabilities of the two events (alive/dead). So there is some intuition behind this definition. The problem (as with the second interpretation) is that we should not bundle everything into one quantity. We still need to separately and explicitly know (or assume equal) $P(L)$ and $P(D)$ and we also need to separately know (or assume equal) $P(A_L|L), P(A_D|D)$. If we have this knowledge (or assumptions) we can then calculate $P(L|A_L,B_L)$. We will just need a couple more algebraic manipulations to derive $P(A_L|L), P(A_D|D)$ from this alternatively-defined "accuracy".

In summary, we could define accuracy in various ways, but when it comes to computing $P(L|A_L,B_L)$, we need to know the quantities popping up in Bayes rule. The more straightforward/natural approach is to equate accuracy with sensitivity and specificity.

The solution

Let's use the accepted norms of sensitivity and specificity and let's make the (straightforward) assumption that they are both equal to the number given to us as "accuracy" (even though we should not be called to make this assumption, the problem description should be explicit). As you have discovered we also need to know the priors. Again the problem description should give you these, but in the absence of any info we can assume that the priors are equiprobable ($P(L) = P(D) = \frac12$). In fact, this is what you have assumed (maybe without realising) in your first solution. Your second solution employs Bayes rule and makes the use/need of the priors explicit. Notice that if you set aside the unconventional definition of accuracy of the second solution, your two solutions are identical.

$$ \begin{align} P(L|A_L,B_L) &= \frac{P(A_L,B_L|L)\cdot P(L)}{P(A_L,B_L)} \\ \\ &= \frac{P(A_L,B_L|L)\cdot P(L)}{P(A_L,B_L|L)\cdot P(L)+P(A_L,B_L|D)\cdot P(D)}\\ \text{(A and B are independent)}\rightarrow \\ &= \frac{P(A_L|L)\cdot P(B_L|L)\cdot P(L)}{P(A_L|L)\cdot P(B_L|L)\cdot P(L)+ P(A_L|D)\cdot P(B_L|D)\cdot P(D)} \\ \\ &= \frac{p_1\cdot p_2 \cdot P(L)}{p_1\cdot p_2 \cdot P(L) + (1-p_1)\cdot (1-p_2) \cdot P(D)}\\ \text{(equiprobable priors)}\rightarrow\\ &= \frac{p_1\cdot p_2 \cdot 0.5}{p_1\cdot p_2 \cdot 0.5 + (1-p_1)\cdot (1-p_2) \cdot 0.5}\\ \\ &= \frac{p_1\cdot p_2}{p_1\cdot p_2 + (1-p_1)\cdot (1-p_2)} \end{align}$$


I always find it useful to visualise these quantities and relationships. I created the following graphic to visualise the whole event space. The three "columns" in this graphic denote the different simple/individual events that can happen: 1) the cat actually being dead or alive, 2) A's prediction that the cat is dead or alive, and 3) B's prediction that the cat is dead or alive. To sample the space you simply take a horizontal line across the columns. For example, line $E$ drawn in the graphic, shows the combination of events where: 1) the cat is alive, 2) witch $A$ predicted alive, and 3) witch $B$ predicted dead. The tricky part is drawing the columns in such a way so that they reflect the independences and correlations between the simple events. For example B predicting true does not depend on A's prediction. Can you see how this is reflected in the drawing? Or note that A's and B's predictions depend on whether the cat is actually alive or dead (linked to their prediction accuracies). Once we have the graphic we can easily visualise the solution. In the graphic, I made the priors not equal (just to stress the importance of their knowledge), I made A's accuracy (= sensitivity = specificity) around $75\%$ and B's accuracy (= sensitivity = specificity) around $55\%$. We can see how the result derived by Bayes rule is the same as the one derived graphically.

Cat dead or alive probability space

I hope this is useful.

$\endgroup$
8
  • $\begingroup$ When you say $P(L|A_L)$ depends on $P(L)$, do you mean that since $P(L|A_L)\le P(L)$, therefore it cannot act as an accuracy, because accuracy could be any value from 0 to 1 $\endgroup$ – fizis Dec 23 '16 at 2:09
  • $\begingroup$ I understood, the prediction of witches should not affect the probability of that cat is dead or alive, therefore $P(L|A_L)=P(L)$. $\endgroup$ – fizis Dec 23 '16 at 2:19
  • $\begingroup$ I am sorry that I haven't select your answer before the bounty expires, this is a good answer, although I am still reading it. $\endgroup$ – fizis Dec 23 '16 at 2:22
  • $\begingroup$ @fizis No I just mean that $P(L|A_L)$ depends on $P(L)$. Note that $P(A_L|L)$ does not depend on $P(L)$. Also it is not true that $P(L|A_L)\le P(L)$, I am not sure how you derived that. $\endgroup$ – Thanassis Dec 23 '16 at 2:22
  • 1
    $\begingroup$ The graph is very intuitive, very good illustration. $\endgroup$ – fizis Dec 23 '16 at 6:46
6
$\begingroup$

If the witches make correct predictions with probabilities $p_1$ and $p_2$ respectively, and their predictions' correctnesses are independent, then a Bayesian analysis gives $$ P(L|L_1\wedge L_2)=\frac{P(L_1\wedge L_2 | L)P(L)}{P(L_1\wedge L_2)}=\frac{P(L_1\wedge L_2 | L)P(L)}{P(L_1\wedge L_2 | L)P(L) + P(L_1\wedge L_2 | D)P(D)}=\frac{p_1p_2P(L)}{p_1p_2P(L)+(1-p_1)(1-p_2)P(D)}; $$ so, assuming a neutral prior ($P(L)=P(D)=1/2$), the cat lives with probability $$ \frac{p_1 p_2}{p_1p_2 + (1-p_1)(1-p_2)}, $$ just as in the first solution. This is probably (IMO) the most natural interpretation of the question. Your definition of "accuracy" in the second solution is insufficient, as it leaves open the possibility that, e.g., the witches are correct more often with live cats. You need the definition to be $$p_i\equiv P(L_i|L)\equiv P(D_i|D).$$

$\endgroup$
2
  • $\begingroup$ Then do you think the first solution is logically right? If not, what's wrong with it? $\endgroup$ – fizis Dec 13 '16 at 2:49
  • $\begingroup$ I agree that this is the answer, but OP's main concern is the definition of accuracy. You define it in the more natural sense, but I feel you do not provide enough explanation/insight on why this the case. Your comment that the second definition of accuracy could lead to witches being correct at different rates for dead cats compared to alive cats is true, but not a reason to dismiss this interpretation/definition IMO. $\endgroup$ – Thanassis Dec 22 '16 at 23:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.