24
$\begingroup$

I'm reviewing some notes regarding probability, and the section regarding Conditional Probability gives the following example:

$P(X,Y|Z)=\frac{P(Z|X,Y)P(X,Y)}{P(Z)}=\frac{P(Y,Z|X)P(X)}{P(Z)}$

The middle expression is clearly just the application of Bayes' Theorem, but I can't see how the third expression is equal to the second. Can someone please clarify how the two are equal?

$\endgroup$
27
$\begingroup$

We know $$P(X,Y)=P(X)P(Y|X)$$ and $$P(Y,Z|X)=P(Y|X)P(Z|X,Y)$$ (to understand this, note that if you ignore the fact that everything is conditioned on $X$ then it is just like the first example).

Therefore \begin{align*} P(Z|X,Y)P(X,Y)&=P(Z|X,Y)P(X)P(Y|X)\\ &=P(Y,Z|X)P(X) \end{align*} Which derives the third expression from the second.

(However I don't have any good intuition for what the third expression means. Does anyone else?)

$\endgroup$
  • $\begingroup$ Excellent! Thank you! $\endgroup$ – Dongie Agnir Nov 4 '13 at 0:03
  • $\begingroup$ I don't see why this question is so popular. $\endgroup$ – Oscar Cunningham Nov 13 '17 at 19:16
  • $\begingroup$ Seriously, why do I keep getting votes? $\endgroup$ – Oscar Cunningham Dec 21 '17 at 22:09
  • 2
    $\begingroup$ Because nobody shows what to do with all those probabilities when you have variables separated by commas anywhere in the expressions $\endgroup$ – Dmitry Avtonomov Apr 3 '18 at 11:33
  • 1
    $\begingroup$ And you just got another one. Your answer helped me with a similar equation on pg 11 of Gelman's 3rd ed BDAS. Ignoring x (cancelling maybe?) makes it obvious that it's just Bayes: p(theta|x,y) proportional to p(theta|x)*p(y|theta,x). btw, I have a question on SO from 2012 that still gets voted, so, it happens. $\endgroup$ – M T Dec 16 '18 at 16:20
14
$\begingroup$

We have

$$P(X,Y\mid Z) \tag1$$

Considering X and Y as a single event, we call them A. Now we have

$$P(A\mid Z) = P(X,Y\mid Z) \tag2$$

Using the Joint Probabilities Rule, we have

$$P(A,Z) = P(A\mid Z)\times P(Z) \tag3$$

So we can say that

$$P(A\mid Z) = \frac{P(A,Z)}{P(Z)} \tag4$$

We know that

$$P(A,Z) = P(Z,A) \tag5$$

Again using the Joint Probabilities Rule, we have

$$P(Z,A) = P(Z \mid A)\times P(A) \tag6$$

We defined $P(A)$ as the following

$$P(A) = P(X,Y) \tag7$$

Again using the Joint Probabilities Rule, we have

$$P(X,Y) = P(X\mid Y)\times P(Y) \tag8$$

Plugging $(8)$ into $(7)$, we have

$$P(A) = P(X\mid Y)\times P(Y) \tag9$$

Plugging $(9)$ into $(6)$, we have

$$P(Z,A) = P(Z\mid A)\times P(X\mid Y)\times P(Y) \tag{10}$$

Plugging $(10)$ into $(5)$ we have

$$P(A,Z) = P(Z\mid A)\times P(X\mid Y)\times P(Y) \tag{11}$$

Plugging $(11)$ into $(4)$, we have

$$P(A\mid Z) = \frac{P(Z\mid A)\times P(X\mid Y)\times P(Y)}{P(Z)} \tag{12}$$

Plugging $(12)$ into $(2)$, we have

$$P(X,Y\mid Z) = \frac{P(Z\mid A)\times P(X\mid Y)\times P(Y)}{P(Z)} \tag{13}$$

Observe that in $(13)$, using the Joint Probabilities Rule, we have

$$P(X,Y) = P(X\mid Y)\times P(Y) \tag{14}$$

Since we defined $P(A)$ is $P(X,Y)$, we have

$$P(A) = P(X\mid Y)\times P(Y) \tag{15}$$

Plugging $(15)$ into $(13)$, we have

$$P(X,Y\mid Z) = \frac{P(Z\mid A)\times P(A)}{P(Z)} \tag{16}$$

Observe that in $(16)$, using the Joint Probabilities Rule, we have

$$P(Z\mid A) = \frac{P(Z,A)}{P(A)} \tag{17}$$

Plugging $(17)$ into $(16)$, we have

$$P(X,Y\mid Z) = \frac{P(Z,A)}{P(Z)} \tag{18}$$

Now observe the following

$$P(Z,A) = P(Z,X,Y) = P(Y,Z,X) \tag{19}$$

Similar to what we did at the beginning, treating $Y$ and $Z$ as a single event and using the Joint Probabilities Rule, we have

$$P(Y,Z,X) = P(Y,Z\mid X)\times P(X) \tag{20}$$

Plugging $(20)$ into $(19)$, we have

$$P(Z,A) = P(Y,Z\mid X)\times P(X) \tag{21}$$

Plugging $(21)$ into $(18)$, we have

$$P(X,Y\mid Z) = \frac{P(Y,Z\mid X)\times P(X)}{P(Z)} \tag{22}$$

I don't know if this clarifies or complicates things more but nevertheless I wanted to include this here as well.

Right now, I can't prove why treating multiple joint events as if they were a single event is "legal".

$\endgroup$
  • $\begingroup$ Please edit your answer by adding $\rm\LaTeX$, for some basic information about writing math at this site see e.g. here, here, here and here. $\endgroup$ – Hakim Oct 26 '14 at 12:58
  • $\begingroup$ Can you explain why (5) P(A,Z) = P(Z,A). I don't understand why this is the same $\endgroup$ – Mr Bonjour Feb 5 '15 at 10:20
  • $\begingroup$ @MrBonjour Oh, actually I don't have a rigorous proof for that but it simply states that: "Probability of occurrence of A AND B together is the same as probability of occurrence of B AND A together.". As I said, I don't have a rigorous proof for it but both sides of the equation state: "The probability of occurrence of two (given) events at the same time.". Since they are occurring at the same time, I don't think that specifying an order to this has a meaning. So this might be a casual explanation for that. $\endgroup$ – Utku Feb 5 '15 at 10:54
  • $\begingroup$ Ok thank you. Actually I'm agree with your explanation. By the way your previous example give me a good grasp of the equation. Thank you again $\endgroup$ – Mr Bonjour Feb 6 '15 at 8:16
  • 1
    $\begingroup$ P(A,Z)=P(Z,A) because events Z and A are independent, not because they are simultaneous or not. If A and Z are not independent, then this equality is not true. Example of not independent events that occur simultaneously: A = encounter a person with long hair, Z = encounter a woman $\endgroup$ – Dmitry Avtonomov Apr 3 '18 at 11:43
1
$\begingroup$

It is easy to follow the following argumentation $$P(Z|X,Y)P(X,Y) = \frac{P(X,Y,Z)}{P(X,Y)}P(X,Y) = P(X,Y,Z)=\frac{P(X,Y,Z)}{P(X)}P(X)=P(Y,Z|X)P(X).$$

$\endgroup$
  • 10
    $\begingroup$ you are being unnecessarily tactless. we are here to help and get help, not to show off.. $\endgroup$ – Giovanni Sep 7 '15 at 6:19
  • $\begingroup$ I have to say that I found this explanation the easiest to follow, even so for the original, one-line version preceding the edit. Clearly, some folks might have taken offense with @royli prefacing his answer with "Isn't this obvious?" but for me, when explained that way, it was obvious! $\endgroup$ – Robert Apr 21 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.