3
$\begingroup$

I am bit puzzled by using gradient descent method. My doubt is that gradient descent is an iterative method for finding minima/maxima of a cost landscape. And it uses steepest ascent/descent method. But on the other hand setting my gradient vector equal to zero vector I should have got my optimal point. Then why to use gradient descent??

$\endgroup$
  • $\begingroup$ I think it's best to think of a classical parabola. If you are far away from the optimum the large gradient pushes you fast towards the optimal point. If you are in the vicinity of the optimal point, the gradient descend is a good guess on how close the real optimum is. The method should converge in the optimal point and there the gradient is zero. $\endgroup$ – Quickbeam2k1 Oct 7 '13 at 17:30
  • $\begingroup$ Thanks Quickbeam2k1 for reply. I think I am not able to explain you problem well.What you said is true. My doubt is why we need gradient descent. Knowing the equation of parabola and setting its first derivatives to zero I should get optimal points. Isn't that true?? So whole point I want to make is that what is need of gradient descent method?? $\endgroup$ – Chandresh Sharma Oct 7 '13 at 17:48
  • $\begingroup$ The gradient method is just a different method that can work. Furthermore it can be applied to higher dimensional problems (and if i remember correctly even to implicit defined ones). Of course for the parabola the calculation is easy. But what would be the extremum of e.g $e^x +x \log(x) -x$. This is hard to calculate by hand (if not impossible) $\endgroup$ – Quickbeam2k1 Oct 7 '13 at 18:06
  • $\begingroup$ Okay !!! I got your point . Thanks $\endgroup$ – Chandresh Sharma Oct 7 '13 at 18:22
3
$\begingroup$

setting my gradient vector equal to zero vector I should have got my optimal point.

More precisely: setting the gradient vector equal to zero vector and solving this equation you would find a critical point.

One equation between vectors in 2 dimensions is actually a system of 2 equations. Those are in general nonlinear (unless your landscape happens to be a second-degree polynomial, which I doubt). To solve such a thing with algebraic manipulations is unlikely. One would have to use some numerical method for solution, which is probably going to be iterative itself.

In addition, equating the gradient to zero could give you a maximum or a saddle point. The gradient descent is more likely to approach a minimum, though it may be fooled by a saddle point sometimes. One still has the issue of figuring out whether the found minimum is global... but this is an issue for every method.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.