0
$\begingroup$

The following problem is from the Machine Learning book by Bishop, and I don't have a background in doing these kinds of computations.

Let $ \mathcal{N} $ and $ \mathrm{Gam} $ be the standard normal and gamma distributions, $$ \mathcal{N}(x | m, \Lambda) = \left( \frac{\beta}{2\pi} \right)^{N/2} \exp \left\lbrace -\frac{1}{2}(x-m)^T \Lambda^{-1}(x-m)\right\rbrace $$ $$ \mathrm{Gam}(\lambda | a,b) = \frac{b^a}{\Gamma(a)} \lambda^{a-1} e^{-b\lambda} $$

In this discussion let $ \mathbf{x} $ be the list of observations $ x_1, \ldots, x_N $, each of which is in $ \mathbb{R}^D $, corresponding to the "outputs" or "labels" $ \mathbf{t} = \{ t_1, \ldots, t_N \} $, $ t_n \in \mathbb{R} $. Let $ \phi_j : \mathbb{R}^D \rightarrow \mathbb{R} $ be some fixed set of "features" for $ j = 1, \ldots, M $, so that $ \phi(x_n) = (\phi_j(x_n))_{j=1}^{M} \in \mathbb{R}^M $. Suppose our aim is to perform linear regression on the dataset $ (\mathbf{x}, \mathbf{t}) $, i.e. we are to find $ w \in \mathbb{R}^M $ to model the relationship via $$ t = y(x,w) := w^T \phi(x) $$ However, suppose there is noise in the data. That is, the outputs $ \mathbf{t} $ are chosen from the distribution $$ p(\mathbf{t} | \mathbf{x}, w, \beta) = \prod_{n=1}^{N}{\mathcal{N}(t_n | w^T \phi(x_n), \beta)} $$ Suppose our prior belief before observing $ (\mathbf{x},\mathbf{t}) $ was (for some fixed constants subscripted by $0$'s) $$ p(w,\beta) = \mathcal{N}(w|m_0, \beta^{-1}S_0) \mathrm{Gam}(\beta|a_0,b_0) $$ I need to compute my posterior belief $$ p(w,\beta | \mathbf{x}, \mathbf{t}) \propto p(\mathbf{t} | \mathbf{x}, w, \beta) p(w,\beta) $$ According to Bishop, the author, this can be expressed again as $$ p(w,\beta | \mathbf{x}, \mathbf{t}) \propto \mathcal{N}(w | m_N, \beta^{-1}S_N) \mathrm{Gam}(\beta | a_N, b_N) \label{1} \tag{1} $$ for some appropriate constants / vectors / matrices $ a_N, b_N, m_N, S_N $. That is my question, to compute these four quantities. I have only gotten as far as $$ p(w,\beta | \mathbf{x}, \mathbf{t}) $$ $$ \propto \beta^{\frac{1}{2} + a_0 - 1 + \frac{N}{2}} \exp \left\lbrace -b_0 \beta - \frac{\beta}{2}(w-m_0)^T S_0 (w-m_0) - \frac{\beta}{2} \sum_{n=1}^N{(t_n - w^T \phi(x_n))^2} \right\rbrace $$ I'm at a loss. What is the trick one uses to put the exponential back into the form of a Gaussian times a Gamma, \eqref{1}. I'm thinking I should want to express these terms $ (t_n - w^T\phi(x_n))^2 $ in the form $ (w - m)^T \Lambda (w-m) $. Is that right / possible?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.