1
$\begingroup$

I'm trying to maximize the log marginal likelihood of a Gaussian process with respect to its hyper parameters (with a squared exponential kernel, to be specific). I've been referring to the text Gaussian Processes for Machine Learning by Rasmussen & Williams to try to get me through this problem, and I see they refer to the Conjugate Gradient Method very often to maximize the log marginal likelihood, but they don't explain how exactly. Maybe it's obvious, but I'm not seeing it.

The Conjugate Gradient Method solves problems of the form $Ax=b$, which turns out is the same as maximizing $f(x) = 1/2 x ^{T}Ax - x^{T}b$. However, I don't see how either of these equations is particularly relevant to the problem I'm trying to solve. The log marginal likelihood only has one term that's quadratic, so how would one maximize the log marginal likelihood while taking the other terms into account as well? And the covariance matrix is constantly changing with the hyper parameters, not the original data itself -- so I'm not trying to solve for $x$ necessarily in $f(x) = 1/2 x ^{T}Ax - x^{T}b$ which seems to be the goal of the Conjugate Gradient Method?

$\endgroup$
2
$\begingroup$

The conjugate gradient method can also be extended to solve general optimization problems of the form

$\min f(x)$

There are several closely related versions of CG for nonlinear function minimization- the most popular are the Fletcher-Reeves and Polak-Ribierre algorithms.

$\endgroup$
2
  • $\begingroup$ Excellent, thank you so much! I suspected line search was the link, but needed someone to verify CG worked for such cases as well. My peer is under the impression CG would somehow allow us to bypass computing the inverses of covariance matrices when optimizing the log marginal likelihood (which is the reason for our interest in CG), but I'm very skeptical of this as I've seen no evidence of that anywhere yet. Would you happen to know if CG is ever used for this purpose? $\endgroup$ – ucb Jan 27 '17 at 19:48
  • $\begingroup$ Nonlinear CG is often used in maximum likelihood estimation. Without knowing the details of your likelihood function I can't really comment on other approaches that might work well for your particular problem. $\endgroup$ – Brian Borchers Jan 27 '17 at 21:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.