0
$\begingroup$

I am using BFGS and L-BFGS to solve an unconstrained optimization problem. The objective function is the Mean Euclidean Error. The output is given by an Artificial Neural Network. The line search produces step sizes $\alpha$ that satisfy the Wolfe conditions.

The methods struggle to converge to a minimizer $w^*$ when the approximation to the inverse of the Hessian, $H = B^{-1}$, is ill-conditioned, that is it's condition number $k(H) = ||H||_2 ||H^{-1}||_2 = \sigma_1/\sigma_n$ is high.

Here are the plots of gradient norm, relative gap error, step size, condition number:

  • Stop criterion: $||\nabla E|| < 10^{-5}$, as you can see the curves zig-zag a lot.

enter image description here

  • this looks fine

enter image description here

enter image description here

  • condition number increases

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.