1
$\begingroup$

I'm reading a paper, and it has a very simply generative model, which is represented by this probabilistic graphical model.

They calculate the posterior P(A|X) in a way I don't understand, though. It doesn't look like a reformulation of Bayes rule, but maybe I'm wrong. I'm a probability noob so I could be. This is how they calculate it. They are marginilizing over W, but I don't understand how this produces the posterior. How did they derive this formula? How does it work?

Likewise, they then calculate the posterior P(W|X) like this. This look similar but identical, because they're now marginalizing over A, including p(A), whereas before when marginalizing over W, there was no p(W) involved. Is this the same method?

I've studied up on marginalization and I can't put the pieces together. Likewise, I'm familiar with Bayes rule, but I can't see how it's utilised here. Can anyone help me out with an explanation?

Thanks!

$\endgroup$

1 Answer 1

2
$\begingroup$

We begin with Bayes' rule, but we use a proportionality symbol ($\propto$), rather than an equality ($=$). (The constant of proportionality is, of course, $\mathsf P(X=x)^{-1}$.)

$$\begin{align}\mathsf P(A\mid X=x)&=\mathsf P(X=x,A)/\mathsf P(X=x)\\[1ex]&\propto \mathsf P(X=x,A)\end{align}$$

Next, by Law of Total Probabitity.

$$\begin{align}\mathsf P(A\mid X=x)&\propto\sum_{W}\mathsf P(X=x,W,A)\end{align}$$

The rest is factorisation from the DAG, and distributing out the common factor.

$$\begin{align}\mathsf P(A\mid X=x)&\propto\sum_{W}\mathsf P(X=x\mid W)\mathsf P(W\mid A)\mathsf P(A)\\[1ex]&\propto\mathsf P(A)\sum_{W}\mathsf P(X=x\mid W)\mathsf P(W\mid A)\end{align}$$


And likewise

$$\begin{align}\mathsf P(W\mid X=x)&=\mathsf P(X=x,W)/\mathsf P(X=x)\\&\propto \mathsf P(X=x,W)\\&\propto \sum_A\mathsf P(X=x,W,A)\\&\propto\sum_A\mathsf P(X=x\mid W)\mathsf P(W\mid A)\mathsf P(A)\end{align}$$

$\endgroup$
5
  • $\begingroup$ Thanks this is super helpful, concise and clear! Could you elaborate on what you mean by factorising from the graph and distributing out the common factor? I've not come across that process before $\endgroup$
    – HereItIs
    Aug 20, 2020 at 3:02
  • 1
    $\begingroup$ The Directed Acyclic Graph is simply a visual representation of the Bayesian factorisation. Here the graph can be simplified to the nodes of interest as: $\require{enclose}\enclose{circle}A\to\enclose{circle}W\to\enclose{circle}X$, which encodes that $\mathsf P(X=x,W,A)=\mathsf P(X=x\mid W)\,\mathsf P(W\mid A)\,\mathsf P(A)$ . $\endgroup$ Aug 20, 2020 at 3:34
  • 1
    $\begingroup$ "Distributing out the common factor" is basic algebra. Just as $ab+ac= a\,(b+c)$ , then too $\sum_i a b_i= a\sum_i b_i$ $\endgroup$ Aug 20, 2020 at 3:38
  • $\begingroup$ Thanks! I'm reading a book about graphical models so now understand how the DAC is a representation of the factorisation. However, I'm confused why the mu is ignored in this factorisation. Is it simply because it is used in the likelihood P(X=x|W)? ie why is it not $$P(A|X=x) = P(A) \sum_{W,\mu} P(W|A)P(\mu|W)P(X=x|\mu)$$ $\endgroup$
    – HereItIs
    Oct 19, 2020 at 11:48
  • 1
    $\begingroup$ @HereItIs We may 'sum out' the unknown $\mu$ parameter using the Law of Total Probability: $$\sum_\mu \mathsf P(X=x\mid \mu)\mathsf P(\mu\mid W) =\mathsf P(X=x\mid W)$$ $\endgroup$ Oct 19, 2020 at 23:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .