I read the paper Towards Evaluating the Robustness of Neural Networks by Carlini
there is a phrase:
the algorithm Clipped gradient can get stuck in a flat spot where it has increased some component x_i to be substantially larger than the maximum allowed
I don't understand, why Clipped gradient can get stuck in a flat spot?
The full context, page 7:
$$ x + δ \in [0,1]^n $$ 2) Clipped gradient descent does not clip
x_ion each iteration; rather, it incorporates the clipping into the objective function to be minimized. In other words, we replace
f(x + δ)with
f(min(max(x + δ, 0), 1)), with the min and max taken component-wise. While solving the main issue with projected gradient descent, clipping introduces a new problem: the algorithm can get stuck in a flat spot where it has increased some component
x_ito be substantially larger than the maximum allowed. When this happens, the partial derivative becomes zero, so even if some improvement is possible by later reducing
x_i, gradient descent has no way to detect this.