I am bit puzzled by using gradient descent method. My doubt is that gradient descent is an iterative method for finding minima/maxima of a cost landscape. And it uses steepest ascent/descent method. But on the other hand setting my gradient vector equal to zero vector I should have got my optimal point. Then why to use gradient descent??

• I think it's best to think of a classical parabola. If you are far away from the optimum the large gradient pushes you fast towards the optimal point. If you are in the vicinity of the optimal point, the gradient descend is a good guess on how close the real optimum is. The method should converge in the optimal point and there the gradient is zero. – Quickbeam2k1 Oct 7 '13 at 17:30
• Thanks Quickbeam2k1 for reply. I think I am not able to explain you problem well.What you said is true. My doubt is why we need gradient descent. Knowing the equation of parabola and setting its first derivatives to zero I should get optimal points. Isn't that true?? So whole point I want to make is that what is need of gradient descent method?? – Chandresh Sharma Oct 7 '13 at 17:48
• The gradient method is just a different method that can work. Furthermore it can be applied to higher dimensional problems (and if i remember correctly even to implicit defined ones). Of course for the parabola the calculation is easy. But what would be the extremum of e.g $e^x +x \log(x) -x$. This is hard to calculate by hand (if not impossible) – Quickbeam2k1 Oct 7 '13 at 18:06
• Okay !!! I got your point . Thanks – Chandresh Sharma Oct 7 '13 at 18:22