Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It only takes a minute to sign up.
Sign up to join this community
How does one go from
$e^{(-\frac{1}{2\sigma_n^2}(\vec{y} - X^T\vec{w})^T(\vec{y}-X^T\vec{w}))}e^{-\frac{1}{2}\vec{w}\Sigma^{-1}\vec{w}}$
to
$e^{-\frac{1}{2}(\vec{w}- w')^T(\frac{1}{\sigma^2_n}XX^T + \Sigma^{-1})(\vec{w} - w')}$
where $w' = \sigma_n^{-2}(\sigma_n^{-2}XX^T + \Sigma^{-1})^{-1}$
I'm sure the algebra works out upon expansion, but how do you come up with this?
Arise from the grave, oh two year old question!
I was wondering about the same thing, and many people will wonder, as this expression comes up when you calculate posterior in Gaussian process with first exponent being likelihood and the second being prior on the weights. I found this expression in p. 9 of the wonderful book Gaussian Processes for Machine Learning by Rasmussen and Williams and was annoyed that I can't see how to complete squares in multivariate case. If someone could give some intuition, I would be very happy!
Meanwhile, for this answer to be of any use, let me show you a trick I came up with: from the Gaussian prior and Gaussian likelihood we expect Gaussian posterior. Gaussian posterior will have the form $$ e^{-\frac{1}{2}(w - \bar w)^T\Sigma\ (w-\bar w)} $$ with $\bar w$ beeing the mean and $\Sigma$ the covariance matrix. In order to find the mean, we take the derivative of the exponent in the question (there is one transposition missing in victor's post) $$ \left(\frac{1}{\sigma_n^2} (y - X^Tw)^T(y-X^Tw) + w^T\Sigma^{-1}w\right)'_w = \frac{-2}{\sigma_n^2}(y-X^Tw)^TX^T + 2w^T\Sigma^{-1}, $$ and equate it to zero. Solving for $w$ we recover the $\bar w = \sigma_n^{-2}(\sigma_n^{-2}XX^T + \Sigma^{-1})^{-1}$ part.
