In BFGS, we apply a rank-2 update to the approximate inverse Hessian ($H \in \mathbb{R}^{n \times n}$) at each iteration. This update adjusts for the curvature found from the change in gradient and change in step size from the previous iteration. In L-BFGS, we only track adjustments over the previous $m$ updates and approximate the inverse Hessian as $H \approx M^T M$ for some $m \times n$ matrix $M$. In this way, my understanding is that we're disregarding curvature found from early steps, and I'm hoping to understand if this is correct.

If we let $m = k$ where $k$ is our current iteration, will this provide the same update as standard BFGS? If this is the case, as $k$ grows, then I'd imagine it's better to just use full BFGS? Or is there an intuitive explanation of the relationship between L-BFGS and BFGS and how we might recover the BFGS approximate Hessian from current $m < k$ step L-BFGS information?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.