I'm currently teaching myself about second-order optimization in the context of machine learning.
One common application of second-order optimization is minimizing the cost function of say, a neural network or some other machine learning algorithm. This is just an optimization problem in some large nonlinear function, where the parameters to the function (the variables) are the weights and biases in the network.
One common way of doing things is Newton's Method - to repeatedly create quadratic approximations of the nonlinear space using the multivariate Taylor Series to create a local quadratic approximation. Then, you directly minimize that. Then, create another approximation, until you reach the minima of the nonlinear function.
The function that creates the approximation is exactly the multivariate Taylor's theorem, dictated by this:
This is all fine and good, and makes sense.
The problem is that optimizing this directly (by setting gradient to 0) involves taking the inverse of the Hessian, which is computationally expensive. That pure update rule is found by setting the gradient of this approximation w.r.t x as 0, and you get this:
So, many algorithms, like Hessian-Free Optimization and L-BFGS opt to approximate the Hessian, to use some "Quazi-newton Method", where we still continuously make approximations and find the minimum, except this minimum is not calculated exactly.
Often used is either the Steepest Descent or Conjugate Gradients methods of minimizing this approximated function (which avoids directly calculating the inverse hessian). The thing is, all the steepest descent and conjugate gradient things I'm learning about are built to minimize functions of the form
Although it looks VERY similar to the Taylor Expansion quadratic approximation which I'm trying to apply steepest descent / conjugate gradients to, it's slightly different.
a) all the vectors are $x$ instead of $(x-x_0)$ - how do I deal with that?
b) there is a negative sign in front of $b^{T}x$
My question is: Since the use case for steepest descent / conjugate gradients is slightly different than the paraboloid created by the Taylor Expansion, how do I modify steepest descent / conjugate gradients OR the Taylor Expansion approximation to be able to apply conjugate gradients / steepest descent to the approximations created by the Taylor Expansion?
