Stuck with handling of conditional probability in Bishop's "Pattern Recognition and Machine Learning" (1.66)

Question

I've just started working through the book, and I'm stuck with how the author handles conditional probability in (1.66).

The context is as follows. In this chapter we are working with a curve fitting task: we try to fit a polynomial $\sum w_ix^i$ to a training set $\{\mathbf {x}, \mathbf {t}\}$ with an assumption of Gaussian noise, i.e. for a single observed value $t$ for $x$, $p(t|x,\mathbf{w},\beta)=N(t|\sum w_ix^i,\beta^{-1})$, and assuming independence of data points, the likelihood is $p(\mathbf{t}|\mathbf{x},\mathbf{w},\beta)=\prod N(t_n|\sum w_ix^i,\beta^{-1}).$ Prior distribution of $\mathbf{w}$ is a multivariate Gaussian: $p(\mathbf{w}|\alpha)=N(\mathbf{0},\alpha^{-1}\mathbf{I})$.

Now, the author states: "Using Bayes’ theorem, the posterior distribution for w is proportional to the product of the prior distribution and the likelihood function:

$$p(\mathbf{w}|\mathbf{x},\mathbf{t},\alpha,\beta)\propto p(\mathbf{t}|\mathbf{x},\mathbf{w},\beta)p(\mathbf{w}|\alpha) \tag{1.66}$$

Later, this proportionality is used to maximize the probability on the left to obtain MAP value of $\mathbf{w}$, so significant factors cannot be simply omitted on the right.

This is where I'm stuck. The problem is I don't understand how he applies Bayes here. I tried to derive it, and that's what I've got:

$$p(\mathbf{w}|\mathbf{x},\mathbf{t},\alpha,\beta)\propto p(\mathbf{x},\mathbf{t},\mathbf{w},\alpha,\beta)=p(\mathbf{t}|\mathbf{x,w},\alpha,\beta)p(\mathbf{x,w},\alpha,\beta)\tag{A}$$

where the latter equals to $p(\mathbf{w}|\alpha,\beta,\mathbf{x})p(\alpha,\beta,\mathbf{x})$. I see that we can get rid of the second factor here because it is not really interesting if we want to maximize the expression — these are just model parameters or the data which is given. Also, I see that we can rewrite the first factor as $p(\mathbf{w}|\alpha)$. That's why: say we have $p(A|BC)\text{, then } p(A|BC)=\frac {p(ABC)}{p(BC)}=\frac {p(AB)p(C)}{p(B)p(C)}=p(A|B)$, which holds if both ($AB$ and $C$) and ($B$ and $C$) are independent, and indeed both $\mathbf{w}$ and $\alpha$ are independent with both $\beta\text{ and }\mathbf{x}$.

But I don't see why we can "remove" $\alpha$ from the first factor of (A). For one, $\mathbf{w}$ and $\alpha$ are not independent. Maybe I don't understand the problem well enough? Maybe this probability can be factorized so that we can say "this factor is irrelevant,let's hide it under $\propto$"?

So, the questions:

1) Please help with understanding of the proportionality (1.66).

2) It's hard for me to see the benefit of using conditional distributions on things like $\alpha$ and $\beta$. Is, for example, $p(\mathbf{w})$ of any interest in this task? I don't see how any meaning can be attributed to it. Could someone explain that?

3) Is there some common knowledge about conditioning on several random variables, like the one I used ($p(A|BC)=p(A|B)$ if both pairs ($AB$ and $C$) and ($B$ and $C$) are independent), that make it obvious?

Answer 1

Let me answer to these questions in a different order.

3) $p(A|BC)=p(A|B)$ also when $A$ is conditionally independent of $C$ given $B$. Actually, this is taken to be as a definition of conditional independence in that same book, chapter 8.2 "Conditional Independence."

1) The proportionality holds because of 3): in this case it is implicitly assumed that $\mathbf{t}$ is conditionally independent of $\alpha$ given $(\mathbf{x},\mathbf{w},\beta)$.

Moreover, if we look at the the Bayesian network in Figure 8.7, we can see that $\alpha$ and $\mathbf{t}$ are d-separated by a set of nodes $\{\mathbf{x},\mathbf{w},\beta\}$. Indeed, there are only two paths between $\alpha$ and $\mathbf{t}$:

• $\alpha \rightarrow \mathbf{w} \rightarrow \mathbf{t}$ blocked by $\mathbf{w}$ (head-to-tail);
• $\alpha \rightarrow \mathbf{w} \rightarrow t \leftarrow \beta \rightarrow \mathbf{t}$ blocked by $\mathbf{w}$ and $\beta$ (tail-to-tail).

That's it! I think that the author should have at least mentioned why this implicit leaving out of $\alpha$ is justified, even if it is explained further in the text.

2) Seems like it is the root of the Bayesian approach — use probabilities everywhere, adding conditional probability in case of functional dependency, and hope that the joint probability of all involved variables will factorize nicely thanks to conditional independence of some variables.

At this place in the book, I feel that it has a downside that it becomes hard to think of probability spaces of the involved variables in an intuitive way. See that same question in the original, well, question: "what is $p(\mathbf{w})$?"