I am new to neural networks and recently found out about gradient descent.

Something does not sit right with me.


Why does this formula work? Wouldn't it make more sense to have lambda a large value thereby mimizing the cost function?

I am not phrasing my question properly as i am honestly quite confused. How could gradient descent result in a global optimum if it always reduces the value?

  • 1
    $\begingroup$ the lambda controls the descent, you can quickly become unstable and finding the solution will be difficult if not impossible. In a similar way that you have to be careful of step size when solving certain differential equations (namely some nonlinear ones) $\endgroup$ – Chinny84 Nov 26 '15 at 15:05
  • $\begingroup$ If you are looking at minizing the function wouldn't it make sense to have infinity as lambda? And x - infinity would give u negative infinity? $\endgroup$ – aceminer Nov 26 '15 at 15:06
  • 1
    $\begingroup$ You can adjust $\lambda$ using line search. Have a look at en.wikipedia.org/wiki/Line_search $\endgroup$ – Claude Leibovici Nov 26 '15 at 15:08
  • 1
    $\begingroup$ What I am confused about is a case when the loss function actually is not minimized when using a huge learning rate as opposed to a smaller one $\endgroup$ – aceminer Nov 26 '15 at 15:10

Let me explain you clearly:

Learning rate is the length of the steps the algorithm makes down the gradient on the error curve.

So, in case you have a high learning rate, the algorithm might overshoot the optimal point.

And with a lower learning rate, in case of any overshoot, the magnitude of overshoot would be lesser than when you have a higher learning rate.

So, in case of overshoot, you would end up at a non-optimal point whose error would be higher.

  • $\begingroup$ Yes and why isit so? Can I have an example of such a case? Maybe some equations and numbers $\endgroup$ – aceminer Nov 27 '15 at 5:53
  • 1
    $\begingroup$ You might want to have a watch at this lecture of Professor Andrew Ng. He did a great job in explaining the downside of having high learning rate. $\endgroup$ – Dawny33 Nov 27 '15 at 5:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.