0
$\begingroup$

Suppose I have a function $f(x)$ for which I want to find minimums.

I understand that differentiation with respect to $x$ will give direction $+/-$ in $x$ axis to follow in order to minimize.

Choosing a fixed step is understandable, but why do we update by $x'=x - f'(x)\cdot\text{step}$?

$f'(x)$ is the slope and is relevant to $y$ axis.

A more understandable procedure to me is this update $x'=x -\text{sign}(f'(x))\cdot\text{step}$.

Thnaks.

joseph.

$\endgroup$
1
  • $\begingroup$ $step$ is a fraction $\endgroup$ Mar 8, 2015 at 10:13

1 Answer 1

0
$\begingroup$

The algorithms of steepest descent rely on the observation that on a neighbourhood of a point x the function f decrease the most strongly in the oppositive direction of the gradient, $\nabla f(x)$.

The direction $-\nabla f(x)$ is a descent direction of f in x, as $f'(x).d=-||\nabla f(x)||^2<0$, so for a step $\alpha$ sufficiently small $$ f(x - \alpha \nabla f(x) < f(x) $$

Then, if you take a direction of the form $-sign(\nabla f(x))$ it won't be a descent direction anymore.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .