# Projected gradient descent with momentum

Can we apply momentum to projected gradient descent? If so, how should we do that?

In the domain I'm working on, momentum greatly speeds up gradient descent. However, I want to do projected gradient descent, rather than plain-old gradient descent. In project gradient descent, after each update step, we project $x$ to a convex set $\mathcal{C}$. If I apply momentum naively to projected gradient descent, things seem to work poorly. I'm guessing this is because the projection changes the input to the next iteration of gradient descent and the algorithm doesn't expect this.

Is there a suitable update rule for projected gradient descent with momentum? How should we modify the standard rule for momentum to incorporate the projection operation?

• Do a Google search for "Accelerated Proximal Gradient Descent". There is a lot of work in this area. – Michael Grant Jul 21 '16 at 21:28