# Bayes' Theorem with multiple random variables

I'm reviewing some notes regarding probability, and the section regarding Conditional Probability gives the following example:

$P(X,Y|Z)=\frac{P(Z|X,Y)P(X,Y)}{P(Z)}=\frac{P(Y,Z|X)P(X)}{P(Z)}$

The middle expression is clearly just the application of Bayes' Theorem, but I can't see how the third expression is equal to the second. Can someone please clarify how the two are equal?

We know $$P(X,Y)=P(X)P(Y|X)$$ and $$P(Y,Z|X)=P(Y|X)P(Z|X,Y)$$ (to understand this, note that if you ignore the fact that everything is conditioned on $X$ then it is just like the first example).

Therefore \begin{align*} P(Z|X,Y)P(X,Y)&=P(Z|X,Y)P(X)P(Y|X)\\ &=P(Y,Z|X)P(X) \end{align*} Which derives the third expression from the second.

(However I don't have any good intuition for what the third expression means. Does anyone else?)

• Excellent! Thank you! – Dongie Agnir Nov 4 '13 at 0:03
• I don't see why this question is so popular. – Oscar Cunningham Nov 13 '17 at 19:16
• Seriously, why do I keep getting votes? – Oscar Cunningham Dec 21 '17 at 22:09
• Because nobody shows what to do with all those probabilities when you have variables separated by commas anywhere in the expressions – Dmitry Avtonomov Apr 3 '18 at 11:33
• And you just got another one. Your answer helped me with a similar equation on pg 11 of Gelman's 3rd ed BDAS. Ignoring x (cancelling maybe?) makes it obvious that it's just Bayes: p(theta|x,y) proportional to p(theta|x)*p(y|theta,x). btw, I have a question on SO from 2012 that still gets voted, so, it happens. – M T Dec 16 '18 at 16:20

We have

$$P(X,Y\mid Z) \tag1$$

Considering X and Y as a single event, we call them A. Now we have

$$P(A\mid Z) = P(X,Y\mid Z) \tag2$$

Using the Joint Probabilities Rule, we have

$$P(A,Z) = P(A\mid Z)\times P(Z) \tag3$$

So we can say that

$$P(A\mid Z) = \frac{P(A,Z)}{P(Z)} \tag4$$

We know that

$$P(A,Z) = P(Z,A) \tag5$$

Again using the Joint Probabilities Rule, we have

$$P(Z,A) = P(Z \mid A)\times P(A) \tag6$$

We defined $P(A)$ as the following

$$P(A) = P(X,Y) \tag7$$

Again using the Joint Probabilities Rule, we have

$$P(X,Y) = P(X\mid Y)\times P(Y) \tag8$$

Plugging $(8)$ into $(7)$, we have

$$P(A) = P(X\mid Y)\times P(Y) \tag9$$

Plugging $(9)$ into $(6)$, we have

$$P(Z,A) = P(Z\mid A)\times P(X\mid Y)\times P(Y) \tag{10}$$

Plugging $(10)$ into $(5)$ we have

$$P(A,Z) = P(Z\mid A)\times P(X\mid Y)\times P(Y) \tag{11}$$

Plugging $(11)$ into $(4)$, we have

$$P(A\mid Z) = \frac{P(Z\mid A)\times P(X\mid Y)\times P(Y)}{P(Z)} \tag{12}$$

Plugging $(12)$ into $(2)$, we have

$$P(X,Y\mid Z) = \frac{P(Z\mid A)\times P(X\mid Y)\times P(Y)}{P(Z)} \tag{13}$$

Observe that in $(13)$, using the Joint Probabilities Rule, we have

$$P(X,Y) = P(X\mid Y)\times P(Y) \tag{14}$$

Since we defined $P(A)$ is $P(X,Y)$, we have

$$P(A) = P(X\mid Y)\times P(Y) \tag{15}$$

Plugging $(15)$ into $(13)$, we have

$$P(X,Y\mid Z) = \frac{P(Z\mid A)\times P(A)}{P(Z)} \tag{16}$$

Observe that in $(16)$, using the Joint Probabilities Rule, we have

$$P(Z\mid A) = \frac{P(Z,A)}{P(A)} \tag{17}$$

Plugging $(17)$ into $(16)$, we have

$$P(X,Y\mid Z) = \frac{P(Z,A)}{P(Z)} \tag{18}$$

Now observe the following

$$P(Z,A) = P(Z,X,Y) = P(Y,Z,X) \tag{19}$$

Similar to what we did at the beginning, treating $Y$ and $Z$ as a single event and using the Joint Probabilities Rule, we have

$$P(Y,Z,X) = P(Y,Z\mid X)\times P(X) \tag{20}$$

Plugging $(20)$ into $(19)$, we have

$$P(Z,A) = P(Y,Z\mid X)\times P(X) \tag{21}$$

Plugging $(21)$ into $(18)$, we have

$$P(X,Y\mid Z) = \frac{P(Y,Z\mid X)\times P(X)}{P(Z)} \tag{22}$$

I don't know if this clarifies or complicates things more but nevertheless I wanted to include this here as well.

Right now, I can't prove why treating multiple joint events as if they were a single event is "legal".

• Please edit your answer by adding $\rm\LaTeX$, for some basic information about writing math at this site see e.g. here, here, here and here. – Hakim Oct 26 '14 at 12:58
• Can you explain why (5) P(A,Z) = P(Z,A). I don't understand why this is the same – Mr Bonjour Feb 5 '15 at 10:20
• @MrBonjour Oh, actually I don't have a rigorous proof for that but it simply states that: "Probability of occurrence of A AND B together is the same as probability of occurrence of B AND A together.". As I said, I don't have a rigorous proof for it but both sides of the equation state: "The probability of occurrence of two (given) events at the same time.". Since they are occurring at the same time, I don't think that specifying an order to this has a meaning. So this might be a casual explanation for that. – Utku Feb 5 '15 at 10:54
• Ok thank you. Actually I'm agree with your explanation. By the way your previous example give me a good grasp of the equation. Thank you again – Mr Bonjour Feb 6 '15 at 8:16
• P(A,Z)=P(Z,A) because events Z and A are independent, not because they are simultaneous or not. If A and Z are not independent, then this equality is not true. Example of not independent events that occur simultaneously: A = encounter a person with long hair, Z = encounter a woman – Dmitry Avtonomov Apr 3 '18 at 11:43

It is easy to follow the following argumentation $$P(Z|X,Y)P(X,Y) = \frac{P(X,Y,Z)}{P(X,Y)}P(X,Y) = P(X,Y,Z)=\frac{P(X,Y,Z)}{P(X)}P(X)=P(Y,Z|X)P(X).$$

• you are being unnecessarily tactless. we are here to help and get help, not to show off.. – Giovanni Sep 7 '15 at 6:19
• I have to say that I found this explanation the easiest to follow, even so for the original, one-line version preceding the edit. Clearly, some folks might have taken offense with @royli prefacing his answer with "Isn't this obvious?" but for me, when explained that way, it was obvious! – Robert Apr 21 at 14:14