# Naive Bayes: Conditional Independence vs. Marginal Independence Assumption

Apologies if this question would rather belong on Stat SE.

I am following Kevin Murphy's tutorial A brief introduction to Bayes' Rule. I seem to follow his derivations formally, but my intuition in understanding the derivation fails on the following question:

Say we ask "What is the likelihood that someone has a disease given that two tests for this disease yield a positive result?". So, for $D$ for "Disease: yes/no", and $T_1, T_2$ for "Test positive: yes/no":

(1) $P(D=1 \; \vert \; T_1 = 1, T_2 = 1)$.

By the chain rule and the definition of conditional probability, this is equivalent to:

(2) $P(D=1)P(T_1 = 1 \, \vert \, D=1)P(T_2=1 \, \vert \, D=1, T_1=1) \over P(T_1=1, T_2 =1)$

By the Naive Bayes assumption of conditional probability of $T_1, T_2$, given $D$, i.e. $T_1 \bot T_2 \; \vert \; D$, the third term of the numerator can be simplified and we get:

(3) $P(D=1)P(T_1 = 1 \, \vert \, D=1)P(T_2=1 \, \vert \, D=1) \over P(T_1=1, T_2 =1)$

Correct so far? The denominator can be calculated by marginalization, so:

(4) $P(D=1)P(T_1 = 1 \, \vert \, D=1)P(T_2=1 \, \vert \, D=1) \over P(D=1)P(T_1=1,T_2=1 \, \vert \, D=1)\,+\,P(D=0)P(T_1=1,T_2=1 \, \vert \, D=0)$

Applying once more $T_1 \bot T_2 \; \vert \; D$, we get:

(6) $P(D=1)P(T_1 = 1 \, \vert \, D=1)P(T_2=1 \, \vert \, D=1) \over P(D=1)P(T_1=1 \, \vert \, D=1)P(T_2=1 \, \vert \, D=1)\,+\,P(D=0)P(T_1=1 \, \vert \, D=0)P(T_2=1 \, \vert \, D=0)$

Which matches the final step in the Bayes example of the tutorial, and it seems I can in principle follow the derivation up to here.

However, assume now that instead of (4) to (6) above, i.e. marginalization of $P(T_1, T_2)$, then applying conditional independence, we would continue from (3) above as follows, first by deriving the denominator by the chain rule:

(7) $P(D=1)P(T_1 = 1 \, \vert \, D=1)P(T_2=1 \, \vert \, D=1) \over P(T_1=1)P(T_2 =1 \, \vert \, T_1=1)$

Followed by assumption of marginal independence between $T_1$ and $T_2$, i.e. $T_1 \bot T_2$, for:

(8) $P(D=1)P(T_1 = 1 \, \vert \, D=1)P(T_2=1 \, \vert \, D=1) \over P(T_1=1)P(T_2 =1)$

Followed by marginalization, for:

(9) $P(D=1)P(T_1 = 1 \, \vert \, D=1)P(T_2=1 \, \vert \, D=1) \over [P(D=1)P(T_1=1 \, \vert \, D=1) \, + \, P(D=0)P(T_1=1 \, \vert \, D=0)] \; \times \; [P(D=1)P(T_2=1 \, \vert \, D=1) \, + \, P(D=0)P(T_2=1 \, \vert \, D=0)]$

Results (6) and (9) are not identical, my intuition fails what the difference between them is, and more specifically, why (6) is the correct result, and not (9).

Please note: I understand that conditional independence and marginal independence are independent of each other, as well as that my derivation of Naive Bayes is "wrong" in the sense that I am using the wrong kind of independence assumption for the denominator in step (8).

But it seems to me that, intuitively, marginal independence between $T_1$ and $T_2$ should be the "correct" assumption here, in the sense that the numerator expresses the likelihood of the intersection of events $T_1=1, T_2=1$, where we are (implicitly) assuming that these two tests do not "influence or cause each other". The latter is the reason why marginal independence intuitively seems the assumption that should apply here then.

Perhaps someone can explain either where I made a mistake in the above derivations, or why my intuition is wrong here, and why (intuitively) marginal independence for $T_1, T_2$ is not what we want here.

You make the assumption that: $\mathsf P(T_1{=}1,T_2{=}1) = \mathsf P(T_1{=}1)\mathsf P(T_2{=}1)$

However, this is not justified.   The test results are not completely independent for the same individual.   They are independent for random samples from the population.   However, for a specific individual they will only be conditionally independent given the disease state.

Example: Imagine that I have a bag of coins containing only an equal number of double headed and double tailed coins.   If I were to select a coin, toss it, return it, and select another, then I can say the result of the second coin will be independent of the result of the first toss.   However, if I select a coin, toss it, obtain a head, then keep that coin to toss again, I assert that the result of the second toss will be very much dependent on what I obtained on the first toss.

• Thanks you very much for taking the time to answer my question. The marginal vs. conditional independence distinction makes perfect sense of course for your coin example. But I still don't entirely see how and why it applies to the disease+test case. Say I "translate" T_1 as T_{1, for person J}, to state that it is the test result of a particular individual, as you say in your second paragraph. Then P(T_{1, for person J}, T_{2, for person J}) are clearly not marginally independent, as you say. (1/2) – Bert Zangle Oct 25 '15 at 21:41
• But neither does it seem that P(T_{2, for person J} | T_{1, for person J}, D = 1) = P(T_{2, for person J} | D = 1). In other words, if the test result for a specific individual depends on another test result of the same person, it seems this should also prohibit the conditional independence assumption we /are/ allowed to make for Naive Bayes. In other words, I suppose I don't grasp in which sense the (marginal) independence assumption I made is "stronger" than the conditional independence assumption made by the actual Naive Bayes. (2/2) – Bert Zangle Oct 25 '15 at 21:43
• @zanglebert They are the same test, with the same bias when performed independently on the same individual. So their results are conditionally independent given the bias. That bias is just unknown for the selected individual; although it's distribution in the population is. So $$\mathsf P(T_1{=}1, T_2{=}1)= \sum\limits_{d\in\{0,1\}} \mathsf P(T_1{=}1\mid D{=}d)\,\mathsf P(T_2{=}1\mid D{=}d)\,\mathsf P(D{=}d)$$ – Graham Kemp Oct 25 '15 at 22:01
• Think I got it now. Can you let me know if the following correctly expresses the intuitive difference? $\sum\limits_{d\in\{0,1\}} \mathsf P(T_1{=}1\mid D{=}d)\,\mathsf P(T_2{=}1\mid D{=}d)\,\mathsf P(D{=}d)$ is the likelihood of $T_1$ and $T_2$ being positive for a particular individual with a 'fixed' (but unknown) disease value, under the assumption that only the disease value determines the test results (the latter being the consequence of the conditional independence assumption). (1/2) – Bert Zangle Oct 25 '15 at 23:04
• In contrast to: $\mathsf P(T_1{=}1)\mathsf P(T_2{=}1)$ which is the likelihood to receive two positive results from two (independent) samplings, i.e. two different individuals. (2/2) – Bert Zangle Oct 25 '15 at 23:05