# Update step in gradient descent

Suppose I have a function $f(x)$ for which I want to find minimums.

I understand that differentiation with respect to $x$ will give direction $+/-$ in $x$ axis to follow in order to minimize.

Choosing a fixed step is understandable, but why do we update by $x'=x - f'(x)\cdot\text{step}$?

$f'(x)$ is the slope and is relevant to $y$ axis.

A more understandable procedure to me is this update $x'=x -\text{sign}(f'(x))\cdot\text{step}$.

Thnaks.

joseph.

• $step$ is a fraction Mar 8, 2015 at 10:13

The algorithms of steepest descent rely on the observation that on a neighbourhood of a point x the function f decrease the most strongly in the oppositive direction of the gradient, $\nabla f(x)$.
The direction $-\nabla f(x)$ is a descent direction of f in x, as $f'(x).d=-||\nabla f(x)||^2<0$, so for a step $\alpha$ sufficiently small $$f(x - \alpha \nabla f(x) < f(x)$$
Then, if you take a direction of the form $-sign(\nabla f(x))$ it won't be a descent direction anymore.