# How to easily describe the meaning of p(a,b) = p(a|b)p(b) = p(b|a)p(a)?

I know that p(a,b) = p(a|b)p(b) = p(b|a)p(a).

But what is the reason for this?

Why don't p(a,b) = p(a|b)p(b|a) since they all involve with each other.

-
If $a=b$ has probability $< 1$ then $p(a\wedge b)=p(a)<1=p(a|b)p(b|a)$. –  Christian Blatter Jun 8 '11 at 10:12

It is easy to show counter-examples that $\Pr(A,B)$ need not be $\Pr(A|B) \Pr(B|A)$: for example, if $A$ and $B$ always happen together, but do not always happen, then $\Pr(A|B)= \Pr(B|A) = 1$ but $\Pr(A,B) \lt 1$.

Now for the conditional probability statement, let's write $A^c$ to be the complement of $A$ etc. There are four mutually exclusive possible outcomes: $A \cap B$; $A^c \cap B$; $A \cap B^c$; and $A^c \cap B^c$. You have

$$\Pr(B) = \frac{\Pr(A \cap B) + \Pr(A^c \cap B)}{\Pr(A \cap B) + \Pr(A^c \cap B^c) + \Pr(A \cap B) + \Pr(A^c \cap B^c)}$$

where the denominator sums to 1, and also have

$$\Pr(A|B) = \frac{\Pr(A \cap B)}{\Pr(A \cap B) + \Pr(A^c \cap B)}$$

so long as the denominator is not $0$, so multiplying them together gives

$$\Pr(A|B) \Pr(B) = \Pr(A \cap B).$$

A similar argument gives $\Pr(B|A) \Pr(A) = \Pr(A \cap B).$

-
This is very clear proof and very easy to understand. Thank you! –  A-letubby Jun 8 '11 at 11:45

Understanding this would be a lot easier with Venn Diagrams. The probability of A & B is that little space in the middle, but the probability of A given B is that little space (since it's the only part of B where A is true) divided by the whole of B (since those are the available possibilities given). That's the basic formula of probability theory; the measure of cases where something is true divided by the measure of all the possible cases under consideration.

EDIT: An even better illustration would be if a machine randomly shot darts at the Venn Diagram there. If you program it to go anywhere with uniform probability, the chance it goes in both A and B is that little space divided by the area of the whole board (or image). But if you program the machine to only shoot darts in the B region, the probability it still ends up in the A region is that little space divided by the area of B.

-

In general, we define $$P(B \mid A) := \frac{P(A \cap B)}{P(A)},$$ whenever $P(A)>0$. Given this definition, we simply multiply both sides by $P(A)$ to get $$P(A \cap B) = P(A)P(B \mid A).$$ In this sense, then, the formula is basically true by definition. In other words, it is essentially an axiom of probability. It is sometimes called the "multiplication rule".

In fact, many people use this rule intuitively, without any knowledge of formal probability theory. For example, my uncle has no problem calculating the probability of being dealt two aces in Texas hold 'em. He just computes (4/52)*(3/51). He is using the above multiplication rule, although he would not recognize the above formula. To him, this is natural, obvious, and automatic. He would never dream of computing $\binom{4}{2}/\binom{52}{2}$, even though it is the same thing. In fact, I think it would take him some time to even understand the idea of counting combinations like that.

For us, as mathematicians, the rule is true because it is an axiom (or, rather, it is true by definition). This is the best mathematical explanation. If you want to explore further the question of why we choose to adopt this axiom/definition and what, if anything, it has to do with the real world, then you must delve into the philosophical interpretations of probability.

The other major, basic axiom of probability is the "addition rule", which states that if $A$ and $B$ are mutually exclusive, then $$P(A \cup B) = P(A) + P(B).$$ These two rules, together with some technical assumptions about how to handle limits, form the entire basis of probability theory.

Technical note: To elaborate on this last comment, one can show that the above "addition rule", together with the assumption of continuity from below, are sufficient to prove that $P$ is a measure. (See, for example, the introduction to Measure Theory and Probability Theory by Athreya and Lahiri.) The "multiplication rule" gives us the definition of conditional probability, which can be used to define independence. Namely, $A$ and $B$ are independent if $P(B)=0$ or $P(A\mid B) = P(A)$.

-
I think you're getting too theoretical. Bayes' Theorem is called a theorem because you can justify it formally and intuitively - that's the task at hand for this question. –  anon Jun 8 '11 at 18:03