I often read about the importance of momentum in machine learning, namely, in neural networks. And as the partial derivative of the cost function w.r.t. to the weights gives us gradient descent, actually the slope of value in loss function, I can not really understand what part momentum would play in this role. If

$$\text{momentum} = \text{mass} \times \text{velocity}$$

I really don't understand what could be mass or velocity with respect to gradient descent.

Is there any simple explanation? What is the relation?

  • $\begingroup$ Continuous-time gradient descent is of the form $$\dot x = - \nabla f (x)$$ where $ - \nabla f (x)$ is the velocity vector. Discrete-time gradient descent is the 1st order discretization of the ODE above. $\endgroup$ – Rodrigo de Azevedo Mar 13 '18 at 18:38
  • $\begingroup$ Generally optimization algorithms can get stuck near local minima or even sometimes stuck near saddles. I guess momentum is a way of describing this not happening. $\endgroup$ – Ian Mar 13 '18 at 18:43
  • $\begingroup$ @RodrigodeAzevedo Thank you. I think I know where you are heading, but please excuse me, because I am not good in notations. We start our model. We compute gradient descent and we calculate the difference compared to the last value of gradient descent. This way we get the direction, a vector yes. But I am still missing the mass factor. I assume, we are looking for a momentum value which we use to multiply (update the weights), however I am missing this mass factor and what it tells us. $\endgroup$ – Stenga Mar 13 '18 at 21:29
  • $\begingroup$ Mass is irrelevant. Assume unit mass. The relevant parameter is the step size, which is the time step in the discretization of the ODE. However, if you're interested in $$\ddot x = -\nabla f (x)$$ then google "heavy ball method". $\endgroup$ – Rodrigo de Azevedo Mar 13 '18 at 21:34
  • $\begingroup$ @RodrigodeAzevedo Thank you. I have been spending whole day with this, still I am missing something. From my understanding we are acutally calculating force, which is time derivative of momentum. And this force depends how we must update learning rate, so divergence or convergence on local minima don't happen? $\endgroup$ – Stenga Mar 14 '18 at 18:42

There are two reasons momentum appears in training ANNs.

  1. It lets the algorithm "power through" flat areas, local minima, and saddle points. Empirically speaking, high dimensional loss surfaces are not pretty and have lots of these; hence, empirically, momentum has been found to be useful.

  2. It speeds up training (again, empirically), probably partly due (1), but also because it seems to set a more appropriate learning rate (faster in homogeneous areas; slower in rougher ones) and helps avoid oscillations.

Personally, I also speculate that it is helpful in regularizing the stochasticity inherent in SGD (i.e. you "keep around" some of the information from the last training batch).

As for the second part of your question (i.e. "what are the mass and velocity?"), normally SGD is written: $$ \theta_t = \theta_{t-1} - \eta\hat{\nabla}\mathcal{L}(\theta_{t-1}) $$ as an update rule. Note that I'm using $\hat{\nabla}$ because we don't have the gradient; we just have an estimator of it. We can think of this as integrating a physical system, where $\theta$ is position and $\hat{\nabla}$ is velocity. I suppose $\eta$ is essentially $\Delta t$.

Let's define a velocity update to be "weighted": $$ v_{t+1} = \alpha v_t + (1-\alpha)\hat{\nabla}\mathcal{L}(\theta_{t-1}) $$ $$ \theta_{t+1} = \theta_t - \beta v_{t+1} $$ so that the instantaneous velocity at time $t+1$ is controlled by the "mass" $\alpha$. When $\alpha$ is high (i.e. 1), we simply use the velocity from last time, i.e. we are entirely driven by momentum. When $\alpha$ is low (i.e. 0), there is no momentum and the "particle" simply follows where the current "force" is pushing. (Note that I would not take the physical analogy too far though...)

Check out this article on momentum.

Disregarding my own advice above, here's my take on the "physics".

Consider a particle with time-varying position $x_t$, velocity $ v_t $, and mass $m$, undergoing an external force $F(x_t,t)$. We have (by $F=ma$): $$ \frac{dx^2_t}{dt^2} = \frac{1}{m}F(x_t,t) $$ which we can rewrite as: $$ \frac{dv_t}{dt} = \frac{1}{m}F(x_t,t)\;\;\;\;\;\&\;\;\;\;\; \frac{dx_t}{dt} = v_t $$ Discretizing the two equations to first-order (well, symplectic Euler, I guess) gives: \begin{align} v_{t+1} &= v_t + \frac{\Delta t}{m}F(x_t,t) \\ x_{t+1} &= x_t + v_{t+1} \Delta t \end{align} Now, just let $x_t=\theta_t$, $\eta=\Delta t$, and set the external force to be $F(x_t,t) = F(\theta)=\gamma\hat{\nabla}\mathcal{L}(\theta)$. Notice that as $m\rightarrow 0$, the contribution of $F$ becomes higher (i.e. momentum doesn't matter), whereas if $m\rightarrow \infty$, $v_t$ becomes a constant for all $t$ (nothing can move a huge neutron star from whatever its current trajectory is, right?).

Also, let $p_t = mv_t$, $ \xi = \eta/m $, and $\kappa = \eta\gamma$. Then: \begin{align} p_{t+1} &= p_t + \kappa\hat{\nabla}\mathcal{L}(\theta_t) \\ \theta_{t+1} &= \theta_t + \xi p_{t+1} \end{align} This is pretty close to the momentum equations above, the main difference being the use of the weighted average. How can we get something more like that?

Well, let's try to add a friction force or drag, i.e: \begin{align} \frac{\partial x_t}{\partial t} &= v_t \\ \frac{\partial v_t}{\partial t} &= \frac{1}{m}\left( F(x_t) - \epsilon v_t \right) \end{align} Discretizing this one gives: $$ v_{t+1} = \left(1-\frac{\Delta t\epsilon}{m}\right)v_t + \frac{\Delta t}{m}F(x_t) \;\;\;\;\;\&\;\;\;\;\; x_{t+1} = x_t + \Delta t v_{t+1} $$ which we can translate into $$ p_{t+1} = \left(1-\frac{\eta\epsilon}{m}\right)p_t + \eta\gamma\hat{\nabla}\mathcal{L}(\theta_t) \;\;\;\;\;\&\;\;\;\;\; \theta_{t+1} = \theta_t + \frac{\eta}{m} p_{t+1} $$ which we can rewrite as $$ p_{t+1} = \alpha p_t + (1-\alpha)\hat{\nabla}\mathcal{L}(\theta_{t}) $$ $$ \theta_{t+1} = \theta_t - \beta p_{t+1} $$ where we have set: $$ \alpha = \frac{\eta\epsilon}{m},\;\;\;\;\; \beta = \frac{\eta}{m},\;\;\;\;\; \gamma = \frac{1}{\eta} - \frac{\epsilon}{m} $$ Notice that this last form is identical to the form of momentum at which we first looked!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.