I'm learning gradient descent method and I saw different (and opposite) things on my referrals.

I have the following function

$$f(x) = 2x^2 - 5x$$

and I have to calculate some iterations of gradient descent from $$x_0 = 1$$. So, I calculate the function at $$x_0$$, the derivative of the function at $$x_0$$ and now I have to apply the formula

$$x_1 = x_0 - \alpha \cdot f'(x_0)$$

Is $$\alpha$$ randomly chosen or do I have to force the formula to $$0$$ value? I'm quite confused.

The way you choose $$\alpha$$ depends, in general, on the information you have about your function. For example, for the function in your example, it is

$$f'(x) = 4x - 5$$

and $$f''(x) = 4$$, so $$f'$$ is Lipschitz continuous with Lipschitz constant $$L=4$$. You should then choose $$a$$ to be smaller than $$1/L$$, so, in this case, $$a<0.25$$.

In general, you might not know $$L$$. Then you have to resort to a linesearch method (e.g., exact linesearch or Armijo's rule).

You can read Chapter 3 in the book of Nocedal and Wright.

According to the Wikipedia article on gradient descent, $$\alpha$$ is a positive real number. You should choose a small $$\alpha$$, such as $$\alpha = 0.1$$ in your case to avoid going past the minimum value.

• $0.1$ can be quite a large number. – Pantelis Sopasakis May 8 at 9:28
• It still works since the minimum point is at $x = -\frac{-5}{2 \cdot 2} = 1.25$. It really depends on the given function. – Toby Mak May 8 at 9:30