I am wondering how I would apply Bayes rule to expand an expression with multiple variables on either side of the conditioning bar.

In another forum post, for example, I read that you could expand $P(a,z \mid b)$ using Bayes rule like this (see Summing over conditional probabilities): $$P( a,z \mid b) = P(a \mid z,b) P(z \mid b)$$

However, directly using Bayes rule to expand $P(a,z \mid b)$ doesn't seem to be the right way to start out:

$$P(a,z\mid b) = { P(b\mid a,z)P(a,z) \over P(a)}$$

  • 1
    $\begingroup$ The first equation is not Bayes' rule, it's just the definition of conditional probability. You don't need Bayes' rule at all, in fact, here. In your second equation, $P(a)$ should be $P(b)$ in the denominator. $\endgroup$
    – Kirill
    Jun 1 '13 at 22:16
  • $\begingroup$ aiqus.com/forum/questions/5627/… $\endgroup$
    – Ankit
    Feb 10 '17 at 12:05
  • $\begingroup$ For the Bayes rule with two conditions P(A| B,C), check math.stackexchange.com/questions/2463896/… $\endgroup$ Nov 5 '20 at 15:10

Note that you didn't apply Bayes' Rule correctly; Bayes' Rule says that:

$$P(X|Y)={P(Y|X)P(X) \over P(Y)}$$

so your denominator should have actually been $P(b)$. Instead, I will use the definition of conditional probability and multiplication rule (which together imply Bayes' Rule):

\begin{array}{cc} P(X|Y) =\dfrac{P(X,Y)}{P(Y)} & (1)\\ P(X)P(Y|X) =P(X,Y)=P(Y)P(X|Y) & (2) \end{array}

Thus, observe that: $$ \begin{array}{r@{}ll} P\big( (a,z) \mid b \big) &= \dfrac{P(a,z,b)}{P(b)} & \text{by (1), where } X=a,z \text{ and } Y=b\\ &= \dfrac{P(z,b)P\big(a \mid (z,b) \big)}{P(b)} &\text{by (2), where } X=a \text{ and } Y=z,b\\ &= \dfrac{P(b)P(z \mid b)P\big(a \mid (z,b) \big)}{P(b)} &\text{by (2), where } X=z \text{ and } Y=b\\ &= P(z \mid b)P\big(a \mid (z,b) \big) \\ &= P\big(a \mid (z,b) \big) P(z \mid b) \\ \end{array} $$ as desired.


FYI, using the chain rule, P(a,z,b) = P(a|z,b)*P(z|b)*P(b)

  • 1
    $\begingroup$ It should be P(a,z|b) $\endgroup$
    – Babai
    Nov 26 '14 at 9:58
  • $\begingroup$ Plus, $P(a,z,b)=P(a,z|b) P(b)\quad\equiv \quad P(a|z,b)P(z,b)$. $\endgroup$
    – nyxee
    Feb 21 '17 at 2:42

Since the op mentioned expand an expression with multiple variables on EITHER side of the conditioning bar., here is my attempt to derive P(a|z,b) using Bayesian rule, assuming that A is a discrete variable:

$$\begin{align} P(A=a1|z,b) &= \frac{P(a_1, z, b)}{P(z,b)} \\ &= \frac{P(a_1,b|z) P(z)}{P(b|z)P(z)} \\ &= \frac{P(a_1,b|z)}{P(b|z)} \\ &= \frac{P(b|a_1,z)P(a_1|z)}{P(b|z)} \\ &= \frac{P(b|a_1,z)P(a_1|z)}{P(b|a_1,z)P(a_1|z) + P(b|a_2,z)P(a_2|z) + ...} \end{align}$$

Compare this with the "single variable" version where

$$\begin{align} P(A=a_1|b) &= \frac{P(a_1,b)}{P(b)} \\ &= \frac{P(b|a_1)P(a_1)}{P(b)} \\ &= \frac{P(b|a_1)P(a_1)} {P(b|a_1)P(a_1) + P(b|a_2)P(a_2) + ..} \end{align}$$

Can we think of Z as a extra "condition" that's added to both the numerator and denominator?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.