Assume a function similar to a saddle, like this. enter image description here

At the origin, the gradient will be 0. However, assume you have a function similar to this one where the gradient of the saddle at the origin is not 0.

Then obviously, the magnitude of the direction of maximum increase is either in the x direction or in the y direction. However, if you add up those two vectors, which is taking the gradient of this function, you get a vector going sideways, which is not in the direction of maximum increase.

So, this is confusing me. The vector addition of the rate of increase the x direction and the rate of increase in the y direction does not equal the direction of highest increase.


The gradient is not the direction of maximum rate of increase. It is the direction of maximum infinitesimal rate of increase, to first order. Said another way, it is the direction of maximum rate of increase of the first-order approximation.

At a point where the gradient is zero or close to zero, the first-order approximation may stop being a good approximation, and so the gradient may stop having anything to do with the actual direction of maximum rate of increase, which may now be mostly controlled e.g. by the second-order approximation (so by the Hessian). This is a known practical issue when applying gradient descent to optimize a function; sometimes you can get stuck in "plateaus."

| cite | improve this answer | |
  • $\begingroup$ I was assuming that the function was a variation of the saddle point which had the saddle point's general shape and increased the most infinitesimally at the origin in the y direction. $\endgroup$ – Goldname Mar 17 '17 at 18:37
  • $\begingroup$ The gradient in this case isn't going to point in the maximum infinitesimal rate of increase. $\endgroup$ – Goldname Mar 17 '17 at 18:37
  • $\begingroup$ @Goldname: at the saddle point itself, the actual best direction to move in is either in the x or the y direction, as you say. This is because at the saddle point the gradient is zero and so one must work with the Hessian, and then everything is controlled by the eigenvectors and eigenvalues of the Hessian. At a slight tilt of the saddle point so that the gradient is not zero, I claim that if you zoom in far enough it stops looking like a saddle point, and the gradient dominates the shape of the saddle point. If you don't zoom in far enough then the Hessian is still important. $\endgroup$ – Qiaochu Yuan Mar 17 '17 at 18:40
  • $\begingroup$ I know that it is 0 at the origin. Take a point slightly to the right of the origin. Are you saying that the highest rate of change when infinitesimally close to the origin is no longer the x and y direction? $\endgroup$ – Goldname Mar 18 '17 at 0:06
  • 1
    $\begingroup$ @user1: "complete nonsense" is too strong, but I would say the top answer is being imprecise. Halfway through it replaces the function with its first-order approximation without saying so. $\endgroup$ – Qiaochu Yuan Jul 11 '18 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.