Here are some high-level intuitions that seem to be folklore in the optimization community:
The gradient descent method is often motivated from a physical point of view, as a 'ball rolling down a hill', or something to that effect.
This is a good high-level analogy, but once you look closer at the details of the algorithm, this point of view doesn't stand up to scrutiny. For example, while the physical picture suggest that the ball accelerates down the hill, no such thing happens in vanilla gradient descent; it is more like Aristotelian physics in that the 'force' creates constant velocity instead of constant acceleration.
- Still, there are more sophisticated variants on the theme of gradient descent which exhibit such acceleration, such as gradient descent with momentum, or damped Newton's method, and some of these are governed by differential equations resembling actual physical scenarios.
My question: has anyone written down a systematic study of physical interpretations of gradient descent and its variants? More precisely, are there ways to cast such algorithms in a physical setting (even if the physics is 'fake' in the sense that it doesn't quite match real-world physics), and are there interesting quantities (e.g. some form of energy) naturally associated with these interpretations?