# Intuitively, why does Bayes' theorem work?

I'm not looking for a cryptic math demonstration. Rather, I'm interested in the intuition behind the theorem that reveals the a posteriori probability, given the prior probability $\times$ the likelihood.

• Would be helpful if you could be a little more "down to earth" here and give a concrete example rather than just "throwing questions in the air" (i.e., "Why does Bayes' theorem work in the following case..."). – barak manos Mar 28 '15 at 15:14

Draw a Venn diagram to help you understand $P(A|B)=P(A\cap B)/P(B)$. Then use this to relate the quantities $P(A|B)$ and $P(B|A)$ algebraically. Let's discuss the first point.

Suppose we have a finite sample space so we can count the number outcomes in each possible "event." To determine $P(A|B)$, we're essentially asking what the probability of getting an outcome in $A$ is if we uniformly at random (for simplicity) pick an outcome in event $B$.

For example, consider a collection of 100 objects. Say 64 are balls and 36 are blocks. Suppose further that among the balls, 24 are red balls and 40 are blue balls. If $A$ is the event of being a red object (we don't know how many red blocks there are, but it won't matter) and $B$ is the event of being a ball, then $P(A|B)$ is the probability of picking a red object given the fact that the object you picked was a ball, or equivalently of picking a red ball out of all of the balls, which will equal the number of red balls divided by the number of balls, or $P(A\cap B)/P(B)=\frac{24}{64}$.

Now, given $P(A|B)P(B)=P(A\cap B)=P(B|A)P(A)$ it shouldn't be hard to finish.

• Thank you, this is very useful! – NLK511 Mar 28 '15 at 15:23
• @NLK511: It also helps to draw the Venn diagram in a rectangular fashion, as one can see depicted in the answer of user2023861 below. – Lee Mosher Sep 16 '15 at 15:29

The prior distribution and the likelihood function (based on data) both contain information about a parameter. Bayes' theorem allows these two kinds and sources of information to put together into a posterior distribution. The combined information from the posterior distribution can be used make inferences about the parameter. A couple of examples illustrate this process.

Screening test for a disease. Suppose we wonder whether a particular person has a disease. The prevalence in the population to which the subject belongs is 2%, so this can be considered as our prior information about the subject. P(Subj has Disease) = .02. A quick and inexpensive,, but imperfect, screening test for the disease is available. Its characteristics are described in conditional probabilities: P(Pos test | Dis) = .99, P(Neg test | No Dis) = .97. Suppose our data is that the subject tests positive.

Then using the elementary form of Bayes' Theorem we can find the posterior probability P(Dis | Pos test) = 0.4024. Some people, focusing on P{Pos test | Dis} = .99) are surprised the posterior probability is so small.

However, the appropriate focus for our purposes is that the data (positive test result) has gone together with the prior probability of 0.02 to give us a posterior probability about 0.40. The screening test is imperfect, but data from it has made a considerable change in our assessment of the subject's probability of disease. A subject with a 40% chance of having a serious disease should be evaluated with further and perhaps more time consuming and expensive tests.

Public Opinion Poll. A newly hired consultant for a political campaign to elect Candidate A feels that the candidate will win, but not overwhelmingly. Suppose her prior distribution on the probability $\psi$ of winning is $Beta(330, 270)$, which has mean 0.55 and 95% of its probability in the interval $(0.51,0.59).$ Then a poll of 1000 randomly selected potential voters shows 620 of them in favor of Candidate A. This is our data and it is reflected in the binomial likelihood function with kernel $\psi^{620}(1-\psi)^{380}$.

Bayes' Theorem melds the prior distribution with the likelihood function encoding the data to give the posterior distribution $Beta(950, 650)$, where multiplying the prior by the likelihood gives the posterior beta parameters $330+620=950$ and $270+380=650.$ The posterior beta distribution has mean about 0.59 and puts about 95% of its probability in the interval $(0.57, 0.62),$ which we take as our posterior probability interval for $\psi$, a somewhat more optimistic outlook for the candidate than given by the prior.

Here again, the information in the prior distribution and the data (as reflected in the likelihood function) have been combined to give a posterior distribution. Very roughly speaking, it is as if the consultant's prior distribution contributed information equivalent to that in a poll of 600 prospective voters of whom 330 favored the candidate.

Note: I have chosen these two examples, so that the math (if you care to carry it through) is quite simple. In some cases, much more computational effort is required to find and use the posterior distribution. But the computation needs to be viewed as a means to an end: to combine the information in the prior with the information in the data in order to make inferences based on both.

Acknowledgment: Numbers and distributions in these examples are the same as for ones in Ch 5 & 8, respectively, of Suess and Trumbo (2010), Springer.

Start with an example. Say you have a test that discovers some disease. Even if a person tests positive for the disease, all is not lost, since the test may not be accurate.

What are the person's chances of actually having the disease? There are three factors involved:

1. What are the overall chances of contracting the disease? It would be very unlikely for townsfolk of some small town in Europe who never left it to contract Ebola for instance, irrespective of the test results.
2. What is the test accuracy: Given that a person has the disease, what are the chances that you test positive?
3. What is the test accuracy II: How often does the test give positive results? If it almost always gives a negative result, but in your patient's case it gave a positive result, maybe you should be worried.

Combining these factors gives us Bayes's theorem - Factors 1 and 2 increase the probability and should therefore be multiplied, while Factor 3 decreases the probability, and should be divided:

$$P(\text{Is Sick} |\text{Pos. Test}) =\frac{P(\text{Pos. Test} | \text{Is Sick})\cdot P(\text{Is Sick})}{P(\text{Pos. Test})}$$

• nice explanation, thanks a lot :) – NLK511 Mar 28 '15 at 15:31

The answers here are good, but if you're like me, you learn better with visual aids and actual numbers. I have one for Bayes's Theorem using the same testing idea. Here's the setup:

• You have a population of 100,000 people
• 0.5% of this population use a drug
• We have a test that tells us with 99% accuracy if a person is or is not a drug user
• You tested positive for this drug. What are the odds that you're a user given this information?

Let's set this up in a simple grid:

In the columns, I show counts of users and non-users. We are told that 0.5% of the population of 100,000 use this drug, so there are 500 users and 99,500 non-users.

In the rows, I show counts of test results. I will fill in these values in a moment.

I've highlighted cells inside the table in a certain way:

• Green cells are accurate test results. If a person is a user and the test result is positive, the test is accurate and the cell is green. Likewise, if a person is not a user and the test result is negative, the test is accurate and the cell is green.
• Red cells are inaccurate test results. If a person is a user but the test is negative, this is a false-negative and the cell is red. Likewise, if a person is not a user but the test is positive, this is a false-positive and the cell is red.

Let's start filling in the table. Given that we have 500 users, how many positive and negative test results will we have among this subset of the population? We're told the test is 99% accurate, so that's 495 positive results and 5 false-negatives:

Following the same process for the non-users: given that we have 99,500 non-users, how many positive and negative results will we have among this subset of the population? Again, the test is 99% accurate, so that's 98,505 negative results and 995 false-positives.

Time to analyze the results. We're told that you tested positive for the drug, so let's throw out the results pertaining to negative test results and look at only the "Positive" row in the grid. 1,490 people tested positive for the drug, of which 495 actually are users, and 995 are false-positives. From here it's easy, the probability that you're actually a user given that you're in the "Positive" row is $\frac{495}{1490} = 33.2\%$

Turning this into a formula:

$P(Drug User Given Positive Result) = \frac{P(DrugUser)*N*TestAccuracy}{P(DrugUser)*N*TestAccuracy + P(NotDrugUser)*N*(1 - TestAccuracy)}$

This is exactly the formula I calculated to get 33.2%. Here, N is the 100,000 number that I made up. N cancels out and thus my 100,000 is extraneous information, but I thought it'd be helpful to include it.

With this grid I hope it's clear why, even with a test accuracy so high, your chances of actually being a drug user given a positive test result are so much lower. The number of false-positives is large enough to skew the naive intuition of the result.

• @Canada-Area51Proposal, Thanks. Why did you make a recent edit to my answer? Your edits don't add or elucidate any parts of it. After your edits, we're left with some awkward sentences like In the rows, I show counts of test results that I will fill, after I explain how I highlighted the cells in a certain way There are inconsistencies with how you refer to the four setup steps. And what's the point of changing Turning this into a formula to This can be turned into a formula? All that does is make the sentence passive. A highschool English teacher would scold you for that. – user2023861 Dec 27 '17 at 15:29
• Sorry for any mistake. I corrected the 2 sentences in the blockquotes. I edited because some sentences appear redundant and repeated? – Greek - Area 51 Proposal Dec 28 '17 at 1:08
• @Canada Ultimately it's between you and the answerer, but I think you're well overstepping the norm of reasonable editing of someone else's answer. – Rahul Dec 28 '17 at 1:12

It helps to disambiguate the meaning of "accuracy" more precisely like this Reddit comment, in which there's a typo: "1485" (in "out of 1485 people who test positive") ought be "1495". I rewrote it with whole numbers (rather than 0.5% as the disease rate).

To understand the theorem, you need to understand the vocabulary. "99% accurate" doesn't really give us information about the disease. We ought use the following terms:

Sensitivity - the odds that the test will be positive if you have the disease.

Specificity - the odds that the test will be negative if you lack the disease.

Positive predictive value - the odds that the test will correctly predict you have the disease, if you test positive.

Negative predictive value - the odds that the test will correctly predict you lack the disease, if you test negative.

Our population of 10,000 people has a 1% disease rate. So $$\color{springgreen}{1000}$$ people have the disease, and $$\color{forestgreen}{99,000}$$ don't.

We introduce a test that is 98% sensitive and 99% specific. It will correctly identify $$\color{deepskyblue}{980}$$ of 1000 people with the disease and $$\color{red}{98,010}$$ of 99,000 without the disease. It will incorrectly claim $$\color{red}{20}$$ people ($$= 1000 - 980$$) with the disease don't have it, and $$\color{deepskyblue}{990}$$ people ($$= 99,000 - 98,010$$) without the disease have it.

So out of $$\color{deepskyblue}{1970 \; (= 980 + 990)}$$ people who test positive, 980 have the disease. Thus, our positive predictive value is $$\dfrac{\color{springgreen}{1000}}{\color{deepskyblue}{1970}} = 50.76\%$$.

Out of $$\color{red}{98,030 \; (= 98,010 + 20)}$$ who test negative, $$\color{forestgreen}{99,000}$$ do not have the disease. Thus, our negative predictive value is $$\dfrac{\color{red}{98,030}}{\color{forestgreen}{99,000}} = 99.02\%$$.

In this case, this test is first-rate for determining who lacks the disease. The $$\color{deepskyblue}{1970}$$ who test positive can be tested to confirm they do have the disease, whereas those who tested negative need no further tests.

There are many professions where useful probability estimates are made without explicitly employing Bayes's formula, although the 'calculations' are still all about changing estimates given new information. If you believe in this, then you can accept that a math model can also accurately be employed in some situations.

Example 1: A specialist has been working for almost a year with a patient that has a serious form of cancer. The doctor now estimates that the patient has only a 10% chance of living more than a month. He shares the diagnosis and advises the patient that he can't recommend any further aggressive treatments. So, throughout his interactions with his patient, the specialist observed how events unfolded and was able to make 'soft assessment/estimates' while trying several treatments.

Example 2: A detective is meeting with a murder suspect. From his experience, he asks the suspect an indirect question that could shed light on the investigation. He knows that certain 'weird' responses will bring things into sharper focus. The suspect comes up with an offbeat response that the detective figures an innocent person couldn't even come up with. Yes, just circumstantial evidence, but the detective now has a prime suspect.