# Intuition on the direction of steepest ascent always being orthogonal to the level set of the function

THE QUESTION:

Convince me that when on the surface of a smooth hill, the $$(x,y)$$ direction I should take a tiny step in such that my current height doesn't change is always perpendicular to the $$(x,y)$$ direction I should take a tiny step in so that my height changes by the most.

More Mathematically formulated:

Convince me, intuitively, that the direction of steepest ascent is perpendicular to the level-set of a function.

Convince me, intuitively, that if I"m standing on a smooth hill, the direction of steepest ascent is perpendicular to the direction I should move in so that the height doesn't change at all.

(This section is going to be really long, but just because I want to be helpful to potential responders and explain exactly what I understand and what I don't understand in as much depth as possible. If you read it all, thank you so much!)

I've always had trouble understanding that the gradient is the direction of steepest ascent.

I've seen some excellent answers on this site, like this one...

Why is gradient the direction of steepest ascent?

...and this one...

Gradient of a function as the direction of steepest ascent/descent

...and honestly, most answers seem to answer in the same way: by proving that the dot product of a vector of fixed length with the gradient, which by definition is the change in the function at that point, is maximum when the vector of fixed length (the step) points in direction of the gradient.

That answer is fine...but I've always had a little bit of trouble understanding it.

That's because although the phrase "...take the step that points in the direction of the gradient to maximize the dot product between the step's direction and the gradient..." is mathematically sound, the idea of "the direction" of the gradient isn't something I"m really comfortable with, since I view the gradient as an operator on a vector $$\begin{bmatrix} dx\\ dy \end{bmatrix}$$ that outputs by how much some $$f(x,y)$$ would change at some specific $$(x,y)$$ if we took that "step". It's hard for me to think of the gradient as a vector itself.

So yea, I've never really truly understood the "direction of steepest ascent" of a function.

However, something I DO understand is the level-sets of a function. These are all the $$(x,y)$$ points such that some $$f(x,y)$$ stays constant.

For example, if $$f(x,y)=x+2y$$, then $$(x+2y)=1$$ would be a level-set.

In the picture above, the red plane is $$z=f(x,y)$$, and the green plane is $$(x+2y)=1$$. As you can see, the intersection of the two planes is flat, indicating that $$f(x,y)$$ is constant for all $$(x,y)$$ such that $$(x+2y)=1$$.

Now, say I was standing on that intersection, where $$z=1$$, and I wanted to know which $$(x,y)$$ direction to take a step in so that I didn't move up or down the mountain?

I would need to move in a $$(x,y)$$ direction such that $$(x+2y)$$ stayed constant.

Say I take a tiny step in some arbitrary direction. That step will have an $$x$$ component and a $$y$$ component.

We can represent that tiny step as a vector: $$\begin{bmatrix} dx\\ dy \end{bmatrix}$$.

For whatever tiny amount $$dx$$ that step corresponds to in the $$x$$ direction, $$f(x,y)$$ (my height) will change by $$dx$$, since at that $$(x,y,f(x,y))$$ point I'm standing on on that smooth mountain, $$\frac{\partial f}{\partial x}=1$$.

On the other hand, for whatever tiny amount $$dy$$ that step corresponds to in the $$y$$ direction, $$f(x,y)$$ (my height) will change by $$2dy$$, since at that $$(x,y,f(x,y))$$ point I'm standing on on that smooth mountain, $$\frac{\partial f}{\partial y}=2$$.

In general, at any $$(x,y,f(x,y))$$, the amount by which $$f(x,y)$$ changes when I take a tiny step $$\begin{bmatrix} dx\\ dy \end{bmatrix}$$ is the amount by which it changes due to the component of our step in the $$x$$ direction, which would be $$\frac{\partial f}{\partial x} * dx$$, plus the amount that it changes in due to the component of our step in the $$y$$ direction, which would be $$\frac{\partial f}{\partial x} * dy$$.

In this specific example, the function changes twice as much for any step in the $$y$$ direction than it does for any step in the $$x$$ direction. That means that if I don't want $$f(x,y)$$ to change at all, then for whatever amount I move in the $$y$$ direction, I must move negative twice that amount in the $$x$$ direction, since any fixed amount of movement in the $$y$$ direction corresponds to twice the change in height as does any movement in the $$x$$ direction!

In other words, the direction of my step should be: $$\begin{bmatrix} -2\\ 1 \end{bmatrix}$$.

Let's say I was instead standing at an $$(x,y,f(x,y))$$ point where a tiny step in the $$x$$ direction corresponded to 42 times the change in altitude than a tiny step in the $$y$$ direction did.

In other words, $$\frac{\partial f}{\partial x}=42\frac{\partial f}{\partial y}$$ at that point.

Then, to not change height at all (stay on the level-set), I would want to take a tiny step in the $$\begin{bmatrix} 1\\ -42 \end{bmatrix}$$. I'd want to make sure that my step moves me $$-42$$ times as much in the $$y$$ direction as we do in the $$x$$.

More generally, if I'm standing at some point $$(x,y,f(x,y))$$ on a smooth mountain, the step I should take such that my altitude doesn't change (such that $$f(x,y)$$ doesn't change) should always be $$\begin{bmatrix} +\frac{\partial f}{\partial y}\\ -\frac{\partial f}{\partial x} \end{bmatrix}$$

This makes sense to me - no dot products needed so far!!!!

Now, I know that the direction orthogonal to $$\begin{bmatrix} +\frac{\partial f}{\partial y}\\ -\frac{\partial f}{\partial x} \end{bmatrix}$$ corresponds to taking the negative reciprocal of it.

That is:

$$\begin{bmatrix} \frac{\partial f}{\partial x}\\ \frac{\partial f}{\partial y} \end{bmatrix}$$

AND THAT'S THE DIRECTION OF STEEPEST ASCENT!

In summary, I understand why the "direction of no ascent" is what it is.

If I could somehow intuitively understand that the "direction of steepest ascent" when climbing a mountain is always perpendicular to the direction of no ascent, then I would understand why the gradient is in the direction of steepest ascent.

Thanks!

One more thing...

I tagged this question as a soft question simply because I'm looking for intuitive answers more than mathematical proofs, and it's hard to say whether or not intuitive answers are correct.

Copied and pasted from a comment below...

I'd like to be able to picture myself standing on the surface of a smooth hill, standing over a spot where someone took a bright neon marker and traced out a level-curve on that hill, and picture the hill in such a way that the direction in which the hill is steepest is OBVIOUSLY perpendicular to that hill. And as of now, I just can't! It seems just as plausible that some OTHER direction not perpendicular to that bright yellow level-curve could be the steepest direction instead!

• Good question. "the idea of 'the direction' of the gradient isn't something I'm really comfortable with, since I view the gradient as an operator on a vector..." I wonder if you're not being a bit stubborn in this viewpoint. The gradient operator takes a vector $v$ as input and computes the dot product of $v$ with a certain vector $g$. This vector $g$ is an important part of the picture, and it deserves to have a name and to be understood. The operator viewpoint doesn't mean we have to throw $g$ out the window. If you don't want to call it the "gradient", call it something else. – littleO Jun 11 at 15:21
• @littleO thanks for the input! And I agree completely with you. I understand that the gradient as an operator is, computationally, equivalent to taking the vector $\vec{g}$ and taking its dot product with $\vec{v}$. And I also understand that a dot product between two vectors is maximized when $\vec{v}$ and $\vec{g}$ point in the same direction. It's just that for me personally, since it's just so much easier to think of the "direction of no ascent" than the "direction of steepest ascent", if I understood it from this viewpoint I'd have a much better understanding of $\vec{g}$. – Joshua Ronis Jun 11 at 15:38
• And also, although it seems like it SHOULD make intuitive sense that the direction of steepest ascent is always perpendicular to the direction of no ascent, when I try to explain it to myself intuitively...I can't seem able to! And, "if I can't explain it simply, I don't understand it well enough" (one of those cheesy quotes, I think people say it was einstein). Thanks! – Joshua Ronis Jun 11 at 15:40
• Suppose you're located at $x$, and $u$ is a direction of no ascent at $x$. You move a bit, in a certain direction, hoping to increase the value of $f$ as much as possible. (I like to imagine I'm a mosquito looking for a warm spot in $\mathbb R^3$.) If your displacement vector $v$ has a nonzero component in the direction $u$, then that nonzero component is wasted motion. Any motion in the direction $u$ is not helpful. Instead of moving in the direction $v = cu + w$, you could have increased the temperature just as much (while moving a shorter distance) by just moving in the direction of $w$. – littleO Jun 11 at 16:17
• @littleO yep - that makes sense mathematically (and perhaps intuitively too to a lot of people). However, for some reason, I still can't make a visual connection. I'd like to literally be able to picture myself standing on the side of a hill, standing over a spot where someone took a bright neon marker and traced out a line of a level-curve on that hill, and picture the hill in such a way that the steepest direction is perpendicular to that hill. And as of now, I just can't! – Joshua Ronis Jun 11 at 19:02

I don't know how helpful this will be, it's just the way I sometimes like to picture it.

Since your hill is smooth, it's locally just a plane (more precisely, there exists a tangent plane which is an approximation that is at least quadratically good).

Now take this plane and cut out a small disk where you're standing (it will in general be slanted). Draw its horizontal diameter, which is (a piece of) a level set. If you grab the disk at the points where this diameter intersects the boundary and look at it head-on, being careful to only rotate it about the vertical axis, you may be able to convince yourself that indeed the only possibility is going perpendicular to the diameter.

This is rather vague, I hope it's not completely useless.

• That's perfect! Thanks! That's exactly the sorts of answers I'm looking for - little intuitive ways to think about it like that. Thank you! +1 – Joshua Ronis Jun 11 at 15:43
• What do you mean by "where the diameter intersects the boundary"? The boundary of the disk? – Joshua Ronis Jun 11 at 15:45
• Maybe you can do this: instead of dealing with the entire graph, take two perpendicular lines in the $xy$ plane, perhaps $x=const., y=const.$ Then slice the graph of your function along these two lines which gives you two spaghetti floating in 3d space. You have to "glue" a plane to their intersection as flatly as possible. If you look at the spaghetti really close, they're basically lines. Maybe this can help visualize why there's only one tangent plane. – J_P Jun 11 at 16:23
• Yeah, it's really not the right intuition that the tangent plane is a plane that just touched the hill in one point. Rather the tangent plane is the plane that is the same shape as the hill, up to trivial errors, when you zoom in close enough. That presentation makes it clear that there's only one tangent plane. – Kevin Carlson Jun 11 at 16:30
• It doesn't necessarily. This means that the partial derivatives exist but that the function is not differentiable. Actually, looking back at this, I think the way Kevin Carlson put it is clearest. For a function to be differentiable, this means that if you keep zooming in ad infinitum, if you're a tiny tiny speck on the graph, you won't even be able to perceive any curvature at all and the graph will look just like a plane. That plane that you "see" is the tangent plane. – J_P Jun 15 at 22:08