In Bishop's Pattern Recognition and Machine Learning (ISBN-13: 978-0387-31073-2), Bishop writes on page 86:

This is an example of a rather common operation associated with Gaussian distributions, sometimes called ‘completing the square’, in which we are given a quadratic form defining the exponent terms in a Gaussian distribution, and we need to determine the corresponding mean and covariance. Such problems can be solved straightforwardly by noting that the exponent in a general Gaussian distribution $N(x|\mu,\Sigma)$ can be written: $$ -\dfrac{1}{2}(x-\mu)^T \Sigma^{-1} (x-\mu) = -\dfrac{1}{2}x^T \Sigma^{-1} x + x^T \Sigma^{-1} \mu + \text{const}$$

Where $\mu$ is the mean of a multivariate Gaussian and $\Sigma$ is the covariance matrix.

How was the above equation derived? Is there a name for this technique (Looking up completing the square doesn't yield anything)?

  • $\begingroup$ Do understand the "complete the square" technique, do you know how to use it to solve the (scalar) quadratic equation $a x^2 + b x +c=0$ ? $\endgroup$ – leonbloy Jun 15 '15 at 14:22

The left hand side is the density of a multivariate Gaussian distribution.

To get to the right hand side, since this is linear algebra:

$$-\dfrac{1}{2}(x-\mu)^T \Sigma^{-1} (x-\mu)$$ $$= -\dfrac{1}{2}x^T \Sigma^{-1} x +\dfrac{1}{2}\mu^T \Sigma^{-1} x +\dfrac{1}{2} x^T \Sigma^{-1} \mu - \dfrac{1}{2}\mu^T \Sigma^{-1} \mu$$

and the second and third terms are equal by the symmetry of the covariance matrix $\Sigma$ and so $\Sigma^{-1}$ while the fourth term is a constant.

So if you have a problem which is broadly like the right-hand side, you can transform it into the left-hand side and then use the known properties of a multivariate Gaussian. It is rather like the quadratic completing the square:

$$ax^2+bx+c = a\left(\left(x+\frac{b}{2a}\right)^2 - \left(\frac{\sqrt{b^2-4ac}}{2a}\right)^2\right)$$

  • $\begingroup$ Haha, woops. Thanks. $\endgroup$ – lycus Jun 15 '15 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.