14
$\begingroup$

I understand that the gradient is the direction of steepest descent (ref: Why is gradient the direction of steepest ascent? and Gradient of a function as the direction of steepest ascent/descent).

However, I am not able to visualize it.

enter image description here

The Blue arrow is the one pointing towards minima/maxima. The gradient (black arrow) is not and that's why we have this zig-zag motion.

Then how come gradient is the direction of steepest descent/ascent?

I have a related question: Why does gradient ascent/descent have zig-zag motion?

$\endgroup$
11
  • 40
    $\begingroup$ The water flows by following, at each point, the direction of steepest descent. Thus, every river is a straight line from the spring to the sea, as any good atlas shows. Is it not? $\endgroup$
    – user228113
    May 5, 2017 at 22:47
  • 4
    $\begingroup$ The gradient doesn't point to the function's maximum, but to a direction that locally if you go in that direction the function will change the most $\endgroup$
    – Ofek Gillon
    May 5, 2017 at 22:55
  • 3
    $\begingroup$ @G.Sassatelli That bit of poetry is lost on me. It's clearly not literally true, for at least one reason that water has momentum and therefore rivers change course over the years and not related to steepest descent. Is this some reference to some quote or something? $\endgroup$
    – Todd Wilcox
    May 6, 2017 at 4:58
  • 7
    $\begingroup$ @ToddWilcox: Argumentum ad absurdum. An idealised river (ignoring messy details like momentum) will flow down the locally steepest path at all points. If that were equivalent to always pointing to the global minimum, rivers would be straight lines. As we know they aren't, steepest descent and direct path to the global minimum can't be equivalent. $\endgroup$
    – Tim Pederick
    May 6, 2017 at 5:18
  • 7
    $\begingroup$ If the gradient always pointed to the minimum, then it would also always point away from the maximum. But satisfying both of these constraints (for a function having both one maximum and one minimum) is of course impossibly as soon as min, max, and current position are not collinear. $\endgroup$
    – Hagen von Eitzen
    May 6, 2017 at 12:41

3 Answers 3

Reset to default
25
$\begingroup$

Although the gradient vector is defined at every point, it is really a local concept.

At any given point, it tells you the direction in which the function changes with the greatest rate. If you think of the function as height, then it gives the direction in which the ground is steepest.

As soon as you move an inch, the ground changes and the steepest direction changes.

Instead of the black zig-zag, you need an integral curve of the gradient vector field.

$\endgroup$
3
  • $\begingroup$ Thank you @Fly by Night. Can you please elaborate further on "Instead of the black zig-zag, you need an integral curve of the gradient vector field." $\endgroup$
    – The Wanderer
    May 6, 2017 at 0:49
  • 3
    $\begingroup$ @TheWanderer: Each arrow of the black zig-zag is (or should be) infinitesimally long. Or to put it another way, the black zig-zag is an overly coarse sampling of what would, at high enough resolution, be a smooth curve. $\endgroup$
    – Tim Pederick
    May 6, 2017 at 5:15
  • 1
    $\begingroup$ @TheWanderer At every point there is a gradient vector. An integral curve is a curve that is tangent to these gradient vectors at all of its points. Instead of making big steps like the black zig-zag, move a tiny amount in the direction of the gradient. You've moved so the gradient vector has (probably) changed. Now move a tiny bit in the new direction. Look at where the new vector is pointing then move a tiny bit in that direction, etc. If your steps are small enough then you'll get something that looks like a curve. upload.wikimedia.org/wikipedia/commons/5/5a/Slope_Field.png $\endgroup$
    – Fly by Night
    May 8, 2017 at 18:49
7
$\begingroup$

The gradient $\nabla f(x)$ points in the direction $u$ such that the directional derivative $D_u f(x)$ is as large as possible. You probably walk downhill in the direction of steepest descent, despite the fact that the lowest point on earth is the Dead Sea and you are probably walking in completely the wrong direction to reach it.

Edit:

Maybe walking down a hill is not a perfect analogy because it makes it seem like the "direction of steepest descent" should be a vector in $\mathbb R^3$, with a $z$ component as well as $x$ and $y$ components.

Perhaps a better analogy is a bug walking on a hot (painfully hot!) sidewalk. The bug moves in the direction of steepest descent (the direction in which temperature decreases most quickly), but the bug does not realize that the coolest spot on the sidewalk is ten meters in the opposite direction, where there is shade. Hopefully in this analogy it's clear that the temperature is a function $f(x,y)$, and the direction of steepest descent is a vector with an $x$ component and a $y$ component, but no $z$ component.

$\endgroup$
7
$\begingroup$

Your black lines are not gradient lines at all. The gradient should be perpendicular to the contour lines at every point. Even in an ellipsoidal valley, the gradient will not point to the lowest point, but it will point much closer to it than your picture indicates.

A function minimizer that follows the local gradient has to take a finite sized step in the direction of the gradient, then find the gradient at the new location to take the next step. Often evaluating the gradient is very expensive and you want to do it as few times as possible. One approach is then to follow the gradient from your current point as far as the function stops decreasing, then stop, evaluate the local gradient, and set off in that direction. If that is your strategy, each new direction will be at a right angle to the prior direction. If the new gradient were not perpendicular to the old direction of travel, you could decrease the function by moving farther or not so far in the old direction of travel. You only change direction when you are at a local minimum in the direction you are going.

$\endgroup$
1
  • $\begingroup$ "Even in an ellipsoidal valley, the gradient will not point to the lowest point, but it will point much closer to it than your picture indicates.". --- Even I felt the same way but apparently, if the ellipse is elongated, the gradient points almost perpendicular to the lower point. Please see my earlier question that provide more reference to the beginning of this confusion: math.stackexchange.com/questions/2256925/… $\endgroup$
    – The Wanderer
    May 5, 2017 at 23:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .