0
$\begingroup$

I'm learning gradient descent method and I saw different (and opposite) things on my referrals.

I have the following function

$$f(x) = 2x^2 - 5x$$

and I have to calculate some iterations of gradient descent from $x_0 = 1$. So, I calculate the function at $x_0$, the derivative of the function at $x_0$ and now I have to apply the formula

$$x_1 = x_0 - \alpha \cdot f'(x_0)$$

Is $\alpha$ randomly chosen or do I have to force the formula to $0$ value? I'm quite confused.

$\endgroup$
4
$\begingroup$

The way you choose $\alpha$ depends, in general, on the information you have about your function. For example, for the function in your example, it is

$$ f'(x) = 4x - 5 $$

and $f''(x) = 4$, so $f'$ is Lipschitz continuous with Lipschitz constant $L=4$. You should then choose $a$ to be smaller than $1/L$, so, in this case, $a<0.25$.

In general, you might not know $L$. Then you have to resort to a linesearch method (e.g., exact linesearch or Armijo's rule).

You can read Chapter 3 in the book of Nocedal and Wright.

$\endgroup$
0
$\begingroup$

According to the Wikipedia article on gradient descent, $\alpha$ is a positive real number. You should choose a small $\alpha$, such as $\alpha = 0.1$ in your case to avoid going past the minimum value.

$\endgroup$
  • 2
    $\begingroup$ $0.1$ can be quite a large number. $\endgroup$ – Pantelis Sopasakis May 8 at 9:28
  • $\begingroup$ It still works since the minimum point is at $x = -\frac{-5}{2 \cdot 2} = 1.25$. It really depends on the given function. $\endgroup$ – Toby Mak May 8 at 9:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.