Trouble with gradient intuition

Question

I'm currently learning about gradients, and I thought khanacademy could help me acquiring some intuition. The actual computation is clear to me, however I'm having trouble understand the intuition.

This is the video I'm talking about, and the part I have a part with is 6:11. He says the gradient vector shows the direction you have to travel in the x,y-plane in order to get a maximum slope in the z-direction.

This sounds like gibberish to me. How do you have to 'travel' it? Why do you get a maximum slope? I just have no clue.

Answer 1

At each point in the $xy$-plane the function you have gives you a value. Normally this value is visualized as altitude, that is, a $z$-value, but it can just as easily be anything else. Examples that come to mind are grayscale tones (if the $xy$-plane is a black-and-white picture), density (if the $xy$-plane is a plate made from varying materials), or, maybe the most important one, potential in some force field (often gravitational or electric fields). Of course, in practice, these examples rarely gives you a function that is nice to work with, but in textbook examples it usually works out.

Now, imagine that you are walking around in the $xy$-plane, sniffing or measuring that value, whatever it represents. At some point in time you're standing at the point $(5, 3)$, and the gradient of your function at that point is $(-1, 2)$. That means if you turn around so that you're facing the direction $(-1, 2)$ at the point $(5, 3)$ (that is, you're looking directly at the point $(4, 5)$), then that is the direction where (at least for the first gazillionths of a meter you travel) your value will increase the fastest, out of all the directions you can pick.

I can do even better. I can tell you how much it will grow for that gazillionths of a meter. Since the amplitude of the gradient is $\sqrt 5$, your function value will grow approximately $\dfrac{\sqrt{5}}{\text{gazillion}}$ of whatever unit it's measured in. (Approximately because even over that short distance, the gradient might change a little bit. Studying how these changes affect the end result is what partial differential equations, and especially using computers to solve them, are all about. But that's another story entirely)

---

Traveling around in the $xy$-plane is something we mathematicians do all the time. When you finally get some intuition, it's much easier (and more fun) to play around with that intuition if you imagine yourself being at a point in the plane and walk around, rather than just imagining a point moving around.

---

Also, as for why the gradient gives you a maximal slope, say again that you have a gradient of $(-1, 2)$ at the point $(5, 3)$. That means if you go from that point, and move directly parallel to the $x$-axis (for a very short distance), in the positive direction (towards $(6, 3)$), then your function value will decrease at the speed of $1$ per meter travelled. This is, after all, how the $-1$ in $(-1, 2)$ came to be in the first place. So if you want the value to increase, you're better off moving in the negative $x$-direction.

Likewise, if you move in the positive $y$-direction (towards $(5, 4)$), your function value will increase by $2$ per meter travelled. So you can see that the function grows faster in the positive $y$-direction than in the negative $x$-direction. However, the direction that will give you the fastest growth is a balanced mix of the two. Hence both the directions $(-1, 0)$ and $(0, 1)$ will give you growth, but the fastest growth is the direction $(-1, 2)$. Twice as much in the $y$-direction because the function grows twice as fast that way as in the $x$-direction.

Answer 2

Here's another viewpoint. From single variable calculus, we know that if a function $f$ is differentiable at $x$, then \begin{equation} f(x + \Delta x) \approx f(x) + f'(x) \Delta x \end{equation} and the approximation is good when $\Delta x$ is small.

The situation in multivariable calculus is analogous. Suppose $f:\mathbb R^2 \to \mathbb R$ is differentiable at a point $x = (x_1,x_2) \in \mathbb R^2$. Then \begin{equation} f(x+\Delta x) \approx f(x) + \langle \nabla f(x), \Delta x \rangle \end{equation} and the approximation is good when the vector $\Delta x$ is small.

We could ask, how should we pick $\Delta x$ so that $f(x + \Delta x)$ is as large as possible? We certainly don't want $\Delta x$ to be pointing in the opposite direction as $\nabla f(x)$, because then $\langle \nabla f(x), \Delta x \rangle$ will be negative -- the value of $f$ will decrease! Nor do we want $\Delta x$ to be orthogonal to $\nabla f(x)$, because then $\langle \nabla f(x), \Delta x \rangle = 0$, which doesn't seem to help $f$ get any larger. We want to pick $\Delta x$ to be in the same direction as $\nabla f(x)$.

Answer 3

This graph (click here) tries to give some intuition

1) Vector A is the gradient

2) z=f(x,y) is not shown in the graph

3) Don't miss the point that both angles (theta) are the same!

4) Point represented by the end of vector B is the result of moving in the x and y direction from (1,2) but maintaining the proportion given by the gradient vector. So, if A gives a 2/1 relation –y against x–, then you can remove two units of x (dz/dx=1) and add one unit of y (dz/dy=2), and z will remain the same.

5) The magnitude of vector B is higher that the magnitude of vector A. So, if you take a vector in the B direction but with the magnitude of vector A –that is, smaller than B–, then the positive impact in z due to the variation of y will be lower than the negative impact in z due to the variation of x (the angle formed by the variation of both x and y will be less than theta).

6) Finally, either A' or A'' share the same magnitude than A, but in both cases, the trade off between x and y causes z to go down.

Answer 4

Suppose we have a scalar field (z) in a (x,y) plane –e.g. the temperature of the floor of your room. Then, the gradient is defined by two numbers that can be combined in a single vector (G): (dz/dx, dz/dy). The first number tells how much you will grow in z if you take a little step in the x-direction (suppose dz/dx=1); and the second, how much if we move in the y-direction (suppose dz/dy=2). Then, you have a ratio. For example, 1/2 (variation of z due to variation of x against variation of z due to variation of y). Important now is to realize that any change we made in both x and y, trying to maintain z constant, has to obey the inverse relation; that is, the variation in x has to double the variation in y, because the impact on z of the former is half the impact on z of the latter.

Now, set an origin point and two perpendicular axis on the floor and plot the gradient vector. If you rotate that vector 360º you will cover all the posible movements in the (x,y) plane with the same magnitude (square root of 5). We will need this later.

Define also angle theta as the angle –viewed from (1,2)– formed by the gradient vector and the line x=1. Make sure you understand that we can find the same angle in another place. Go again to point (1,2) and plot from there to the left –until you reach the y-axis– two lines: one perpendicular to the gradient (line P), and the other parallel to the x-axis. As you can check, they also form an angle theta.

Plot also a vector (B) that goes from the origin to any point (C) in the P line.

Finally, the important question: how does z change as we move from (1,2) to C? The answer: no change at all. We have follow the proportion that angle theta dictates: any variation in x will be followed by half of that variation in y. Remember: we are obeying the inverse relation stated by the gradient (impact in z from y doubles impact in z from x), so nothing changes.

But vector B is larger than the gradient vector G... Let's get back to 360º rotation of vector G and choose a vector with the same length than G but pointing in the direction of vector B. This new vector (B') will cross the parallel line to x-axis that we draw before, but will not touch the P line. What does this mean? If we move from point (1,2) to the end of vector B' we will have a higher decrease in x and a smaller increase in y compared to the case where we moved from (1,2) to C. In other words, z will drop.

That's all: if you do not respect the gradient when you move, you will not achieve the highest value in z.

Hope it helps!

Answer 5

Gradient intuition. I had problems too trying to visualize intuitively the gradient in R 3.

I want to share how I finally got it.

For simplicity let us consider a function valued at x=0, y=0 that takes a value z=0 at that point. The shape of this function does not matter because we will look at the tangent plane at x=0, y=0, z=0. Let us also assume that the partial derivative with respect to x at x=0 is 1 and that the partial derivative with respect to y is also 1. Let us now look at the tangent plane defined by the two partial derivatives. (The tangent lines). The partial derivative with respect to x will bisect the z x plane with a slope of 45 degrees. Similarly, the partial derivative with respect to y will bisect the z y plane with a slope of 45 degrees too.
So these two lines define our tangent plane. This tangent touches our z function at z=0.

Now imagine to project a plane, parallel to the x y plane that that intersects the z axis at z=1.

This plane parallel to the x y plane will intersect the tangent plane ( the one mentioned above defined by the two partial derivatives). The intersection is a line. This line passes through the points x=0, y=1, z=1 and x=1, y=0, z=1. Now let us project this line on the x y plane. This projection is the contour line for z=1 on the x y plane. This contour line (A) passes through the points x=1, y=0, z=0 and x=0, y=1, z=0.

Now, what is the shortest direction to take from the origin (the point where the tangent plane touches our function) to reach on the the x y plane the contour line for z=1 described above (A)? It is obvious that the direction is 45 degrees, or the components of the the vector describing the direction are <1,1>. Same as the partial derivatives. One can see that this vector is perpendicular to the contour line on the x y plane.

Now assume that the partial derivative with respect to y is 2, while the partial derivative with respect to x remains 1.

The tangent plane will shift. The plane parallel to the x y plane and with z= 1 will intersect the new tangent plane at points x=1, y=0, z=1 and x=0, y=1/2, z=1.

The projection of this line on the x y will give us the new contour line for z=1.

This new contour line (B) will pass through the points x=1, y=0, z=0 and x=0, y=1/2 z=0.

What is the direction to take now to reach from the origin this new contour line (B) on the x y plane? The vector must be perpendicular to the new contour line (B) and so has components <1,2>, same numbers as the partial derivatives.