I am using BFGS and L-BFGS to solve an unconstrained optimization problem. The objective function is the Mean Euclidean Error. The output is given by an Artificial Neural Network. The line search produces step sizes $\alpha$ that satisfy the Wolfe conditions.

The methods struggle to converge to a minimizer $w^*$ when the approximation to the inverse of the Hessian, $H = B^{-1}$, is ill-conditioned, that is it's condition number $k(H) = ||H||_2 ||H^{-1}||_2 = \sigma_1/\sigma_n$ is high.

Here are the plots of gradient norm, relative gap error, step size, condition number:

  • Stop criterion: $||\nabla E|| < 10^{-5}$, as you can see the curves zig-zag a lot.

enter image description here

  • this looks fine

enter image description here

enter image description here

  • condition number increases

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.