# why use a small learning rate in gradient descent

I am new to neural networks and recently found out about gradient descent.

Something does not sit right with me.

x←x−λ∇fk(x)


Why does this formula work? Wouldn't it make more sense to have lambda a large value thereby mimizing the cost function?

I am not phrasing my question properly as i am honestly quite confused. How could gradient descent result in a global optimum if it always reduces the value?

• the lambda controls the descent, you can quickly become unstable and finding the solution will be difficult if not impossible. In a similar way that you have to be careful of step size when solving certain differential equations (namely some nonlinear ones) – Chinny84 Nov 26 '15 at 15:05
• If you are looking at minizing the function wouldn't it make sense to have infinity as lambda? And x - infinity would give u negative infinity? – aceminer Nov 26 '15 at 15:06
• You can adjust $\lambda$ using line search. Have a look at en.wikipedia.org/wiki/Line_search – Claude Leibovici Nov 26 '15 at 15:08
• What I am confused about is a case when the loss function actually is not minimized when using a huge learning rate as opposed to a smaller one – aceminer Nov 26 '15 at 15:10