I often read about the importance of momentum in machine learning, namely, in neural networks. And as the partial derivative of the cost function w.r.t. to the weights gives us gradient descent, actually the slope of value in loss function, I can not really understand what part momentum would play in this role. If

$$\text{momentum} = \text{mass} \times \text{velocity}$$

I really don't understand what could be mass or velocity with respect to gradient descent.

Is there any simple explanation? What is the relation?

• Continuous-time gradient descent is of the form $$\dot x = - \nabla f (x)$$ where $- \nabla f (x)$ is the velocity vector. Discrete-time gradient descent is the 1st order discretization of the ODE above. Commented Mar 13, 2018 at 18:38
• Generally optimization algorithms can get stuck near local minima or even sometimes stuck near saddles. I guess momentum is a way of describing this not happening.
– Ian
Commented Mar 13, 2018 at 18:43
• @RodrigodeAzevedo Thank you. I think I know where you are heading, but please excuse me, because I am not good in notations. We start our model. We compute gradient descent and we calculate the difference compared to the last value of gradient descent. This way we get the direction, a vector yes. But I am still missing the mass factor. I assume, we are looking for a momentum value which we use to multiply (update the weights), however I am missing this mass factor and what it tells us. Commented Mar 13, 2018 at 21:29
• Mass is irrelevant. Assume unit mass. The relevant parameter is the step size, which is the time step in the discretization of the ODE. However, if you're interested in $$\ddot x = -\nabla f (x)$$ then google "heavy ball method". Commented Mar 13, 2018 at 21:34
• @RodrigodeAzevedo Thank you. I have been spending whole day with this, still I am missing something. From my understanding we are acutally calculating force, which is time derivative of momentum. And this force depends how we must update learning rate, so divergence or convergence on local minima don't happen? Commented Mar 14, 2018 at 18:42

There are two reasons momentum appears in training ANNs.

1. It lets the algorithm "power through" flat areas, local minima, and saddle points. Empirically speaking, high dimensional loss surfaces are not pretty and have lots of these; hence, empirically, momentum has been found to be useful.

2. It speeds up training (again, empirically), probably partly due (1), but also because it seems to set a more appropriate learning rate (faster in homogeneous areas; slower in rougher ones) and helps avoid oscillations.

Personally, I also speculate that it is helpful in regularizing the stochasticity inherent in SGD (i.e. you "keep around" some of the information from the last training batch).

As for the second part of your question (i.e. "what are the mass and velocity?"), normally SGD is written: $$\theta_t = \theta_{t-1} - \eta\hat{\nabla}\mathcal{L}(\theta_{t-1})$$ as an update rule. Note that I'm using $\hat{\nabla}$ because we don't have the gradient; we just have an estimator of it. We can think of this as integrating a physical system, where $\theta$ is position and $\hat{\nabla}$ is velocity. I suppose $\eta$ is essentially $\Delta t$.

Let's define a velocity update to be "weighted": $$v_{t+1} = \alpha v_t + (1-\alpha)\hat{\nabla}\mathcal{L}(\theta_{t-1})$$ $$\theta_{t+1} = \theta_t - \beta v_{t+1}$$ so that the instantaneous velocity at time $t+1$ is controlled by the "mass" $\alpha$. When $\alpha$ is high (i.e. 1), we simply use the velocity from last time, i.e. we are entirely driven by momentum. When $\alpha$ is low (i.e. 0), there is no momentum and the "particle" simply follows where the current "force" is pushing. (Note that I would not take the physical analogy too far though...)

Disregarding my own advice above, here's my take on the "physics".

Consider a particle with time-varying position $x_t$, velocity $v_t$, and mass $m$, undergoing an external force $F(x_t,t)$. We have (by $F=ma$): $$\frac{dx^2_t}{dt^2} = \frac{1}{m}F(x_t,t)$$ which we can rewrite as: $$\frac{dv_t}{dt} = \frac{1}{m}F(x_t,t)\;\;\;\;\;\&\;\;\;\;\; \frac{dx_t}{dt} = v_t$$ Discretizing the two equations to first-order (well, symplectic Euler, I guess) gives: \begin{align} v_{t+1} &= v_t + \frac{\Delta t}{m}F(x_t,t) \\ x_{t+1} &= x_t + v_{t+1} \Delta t \end{align} Now, just let $x_t=\theta_t$, $\eta=\Delta t$, and set the external force to be $F(x_t,t) = F(\theta)=\gamma\hat{\nabla}\mathcal{L}(\theta)$. Notice that as $m\rightarrow 0$, the contribution of $F$ becomes higher (i.e. momentum doesn't matter), whereas if $m\rightarrow \infty$, $v_t$ becomes a constant for all $t$ (nothing can move a huge neutron star from whatever its current trajectory is, right?).

Also, let $p_t = mv_t$, $\xi = \eta/m$, and $\kappa = \eta\gamma$. Then: \begin{align} p_{t+1} &= p_t + \kappa\hat{\nabla}\mathcal{L}(\theta_t) \\ \theta_{t+1} &= \theta_t + \xi p_{t+1} \end{align} This is pretty close to the momentum equations above, the main difference being the use of the weighted average. How can we get something more like that?

Well, let's try to add a friction force or drag, i.e: \begin{align} \frac{\partial x_t}{\partial t} &= v_t \\ \frac{\partial v_t}{\partial t} &= \frac{1}{m}\left( F(x_t) - \epsilon v_t \right) \end{align} Discretizing this one gives: $$v_{t+1} = \left(1-\frac{\Delta t\epsilon}{m}\right)v_t + \frac{\Delta t}{m}F(x_t) \;\;\;\;\;\&\;\;\;\;\; x_{t+1} = x_t + \Delta t v_{t+1}$$ which we can translate into $$p_{t+1} = \left(1-\frac{\eta\epsilon}{m}\right)p_t + \eta\gamma\hat{\nabla}\mathcal{L}(\theta_t) \;\;\;\;\;\&\;\;\;\;\; \theta_{t+1} = \theta_t + \frac{\eta}{m} p_{t+1}$$ which we can rewrite as $$p_{t+1} = \alpha p_t + (1-\alpha)\hat{\nabla}\mathcal{L}(\theta_{t})$$ $$\theta_{t+1} = \theta_t - \beta p_{t+1}$$ where we have set: $$\alpha = \frac{\eta\epsilon}{m},\;\;\;\;\; \beta = \frac{\eta}{m},\;\;\;\;\; \gamma = \frac{1}{\eta} - \frac{\epsilon}{m}$$ Notice that this last form is identical to the form of momentum at which we first looked!

• regarding assertion (2), there is a lot of evidence that there is huge speedup of trainig, but there is also rigorous proof by Nesterov, citing from the article cs.toronto.edu/~fritz/absps/momentum.pdf "In this case, the i-th step of the momentum method will be of distance proportional to i; therefore N steps could traverse a quadratically longer distance: 1 + 2 + · · · + N = O(N^2)" So momentum guaranties the quadratic speedup of the optimization algorithm
– Igor
Commented Jan 24, 2021 at 19:46
• Can you explain what is $\gamma$? Is it equal to $-1$? Commented May 19 at 14:08
• Also why you update the position based on the updated velocity? I.e. why $x_{t+1} = x_t + u_{t+1} \cdot dt$ instead of $x_{t+1} = x_t + u_t \cdot dt$? Commented May 19 at 14:17