4
$\begingroup$

I'm trying to get intuition about why gradient is pointing to the direction of the steepest ascent. I got confused because I found that directional derivative is explained with help of gradient and gradient is explained with help of directional derivative.

Please explain what are the exact steps that lead from directional derivative defined by the limit $\nabla_{v} f(x_0) = \lim_{h\to 0} \frac{f(x_0+hv)-f(x_0)}h$ to directional derivative defined as dot product of gradient and vector $\nabla_{v} f(x_0) = \nabla f(x_0)\cdot{v}$ ?

In other words how to prove the following? $$\lim_{h\to 0} \frac{f(x_0+hv)-f(x_0)}h = \nabla f(x_0)\cdot{v}$$

$\endgroup$
7
$\begingroup$

The limit $\lim_{t\to 0} \frac{f(x_0+tv)-f(x_0)}t$ gives the definition of the derivative in the direction of the unit vector $v$ at $x=x_0\in \mathbb R^n$, that is $\frac{\partial}{\partial v} f (x_0)$.

The formula $$\frac{\partial}{\partial v} f (x_0)=\nabla f(x_0)\cdot v$$ gives a property which is valid under the hypothesis that $f$ is differentiable at $x=x_0$, and is quite useful for calculations. (If $f$ is not differentiable at $x=x_0$, then that relation doesn't need be true, even if all directional derivatives exist.)

The idea of the proof is that being $f$ differentiable at $x_0$, then the gradient $\nabla f(x_0)$ exists and $$\lim_{x\to x_0}\frac{|f(x)-f(x_0)-\nabla f(x_0)\cdot(x-x_0)|}{||x-x_0||}=0$$

Let's think of the point $x=x_0+tv$ (say for fixed $x_0$ and $v$). By definition of directional derivative (and substracting and adding $\nabla f(x_0)\cdot (x_0+tv-x_0$), leads to

$$\frac{\partial}{\partial v} f (x_0)=\lim_{t\to 0} \frac{f(x_0+tv)-f(x_0)}t=$$ $$=\lim_{t\to 0} \frac{f(x_0+tv)-f(x_0)-\nabla f(x_0)\cdot(x_0+tv-x_0)}{||(x_0+tv)-x_0||}\cdot \frac{|t|\,||v||}{t}+\frac{\nabla f(x_0)\cdot(x_0+tv-x_0)}{t}.$$

And because the limit of the first summand is $0$ (why?) (*) and the second one is constant the result is $$\frac{\partial}{\partial v} f (x_0)=\nabla f(x_0)\cdot v,$$ which gives the usual formula.

What might be more interesting to understand this relation is when there's no such relation. Let $f \colon \mathbb R^2 \to \mathbb R$, and $$f(x,y)= \begin{cases} \tfrac{x^2y}{x^2+y^2} & (x,y)\neq (0,0) \\ 0 & (x,y)=(0,0). \\ \end{cases}$$

An easy calculation using the definition shows that, if $v=(v_x,v_y)$ (let's assume $||v||=1$), the directional derivative is in each direction $$\frac{\partial}{\partial v} f (0,0)=\frac{v_x^2 v_y}{v_x^2+v_y^2}=v_x^2 v_y$$ (in particular, both $\frac{\partial}{\partial x} f (0,0)$ and $\frac{\partial}{\partial y} f (0,0)$ are zero, that is $\nabla f(0,0)=(0,0)$.

So, if the 'dot-product formula' were valid, it should be the case that $$\frac{\partial}{\partial v} f (0,0)=(0,0)\cdot (v_x,v_y)=0,$$ which only happens in the directions of the $x$ and $y$ axes. (BTW, this also proves that $f$ is not differentiable at $(0,0)$.)

I suggest you try to imagine why the way in which directional derivatives vary as we change direction in this case (think of the $xy$ plane as the floor) are not compatible with the existence of a tangent plane (differentiability).


(*) In order to verify that $$\lim_{t\to 0} \frac{f(x_0+tv)-f(x_0)-\nabla f(x_0)\cdot(x_0+tv-x_0)}{||(x_0+tv)-x_0||}\cdot \frac{|t|\,||v||}{t}=0,$$ first note that $\frac{|t|\,||v||}{t}$ equals plus or minus $||v||$, depending on the sign of $t$, which means is a bounded function of $t$ ($t\neq 0$). So, to prove our claim is enough to show that $$\lim_{t\to 0} \frac{f(x_0+tv)-f(x_0)-\nabla f(x_0)\cdot(x_0+tv-x_0)}{||(x_0+tv)-x_0||}=0.$$

But this is a consequence of $f$ being differentiable. Indeed, we say that $f\colon \mathbb R^n \rightarrow \mathbb R$ is differentiable at $x_0$ if and only if $$\lim_{x\to x_0} \frac{f(x)-f(x_0)-\nabla f(x_0)\cdot(x-x_0)}{||x-x_0||}=0.$$

Our expression just has $x_0+tv$ instead of $x$, and as the limit is for $t\to 0$, it is also true that $x_0+tv\to x_0$. The only difference is that the definition of differentiable function uses a double/triple/etc. limit (think of sequences of points of $\mathbb R^n$ converging to $x_0$ from every direction and in all sorts of simple or complicated paths), while in our limit $x$ tends to $x_0$ only along the straight line in the direction of $v$. But since $f$ is differentiable at $x_0$, the last limit is $0$, and the same is true if we restrict to the subset of $\mathbb R^n$ that is such line.

$\endgroup$
  • $\begingroup$ Could you please explain the "why?" part? $\endgroup$ – creepyrodent Oct 31 '18 at 17:17
  • $\begingroup$ I added an explanation at the end of the text. I don't know how clear it is, but maybe works for you. $\endgroup$ – Alejandro Nasif Salum Nov 1 '18 at 21:37
2
$\begingroup$

This is really a linear algebra question.

You can show that the directional derivative depends linearly on the direction vector $v$, that is, it satisfies the relation:

$$\nabla_{a v + b u} f = a\,\nabla_v f + b\,\nabla_u f.$$

For scalars $a,b$ and vectors $v,u$. The gradient is the vector of partial derivatives, which in turn are just the directional derivatives in the direction of the basis vectors : $\frac{\partial}{\partial x_k} f =\nabla_{e_k} f$. Now, writing $v$ in the canonical basis $v = v_1 e_1 + \dots + v_n e_n$, by the linearity above :

$$\nabla_{v} f = v_1 \frac{\partial}{\partial x_1} f + \cdots + v_n \frac{\partial}{\partial x_n} f$$

Which is the formula for the dot product you mentioned.

$\endgroup$
  • 1
    $\begingroup$ I agree this is the right perspective: once you know that the directional derivative is linear in the direction, obviously it can be represented by matrix multiplication. But this linearity is a rather deep fact, requiring as Alejandro points out the fact that $f$ is differentiable (i.e. has a local linear approximation). $\endgroup$ – user7530 Jan 16 '18 at 8:49
1
$\begingroup$

This question is best answered by starting from the definition of the derivative.

Let $f: \mathbb{R}^3 \to \mathbb{R}$. (basically the functions you're thinking of in your multivariable class). The good definition of the derivative (read: the one I like and the one that generalizes well to higher dimensions) is the unique linear transformation $L: \mathbb{R}^3 \to \mathbb{R}$ that satisfies:

$$\lim_{h\to 0}\frac{f(x+h) - f(x) - L(h)}{\mid h \mid} = 0.$$

Simply put, the derivative at a point is the linear transform that best approximates the function in a small neighborhood of that point.

Linear algebra tells us that such a linear transform $L: \mathbb{R}^3 \mapsto \mathbb{R}$ is actually given by a matrix (a vector in disguise): $\begin{bmatrix} m_1 & m_2 & m_3 \end{bmatrix}$. Its input h is given by a vector $\begin{bmatrix} h_1 \\ h_2 \\ h_3 \end{bmatrix}$ and calculating $L(h)$ is just a matter of doing matrix multiplication. For this case, it is exactly a dot product between the input and the gradient!

Okay, so what I have shown here is a "general derivative" i.e. a gradient. Now let's find out what a directional derivative is.

It is the same definition,

$$\lim_{h\to 0}\frac{f(x+h) - f(x) - L(h)}{\mid h \mid} = 0.$$

except we restrict $h$ to be proportional to some desired direction vector $v$. This means we "approach" the limit along one specific direction. Then the quantity $L(h)$ is a number that is a directional derivative. If you already know this $L$—remember, this is the gradient—then all you have to do is take a dot product between $L$ and a specific unit vector! This should unify the limit definition and the "dot product definition", and all we needed was a little matrix multiplication (the dot product is matrix multiplication in disguise) and a refinement of the limit definition.


Advanced material

In general (i.e. maps that aren't just from 3-D space to 1-D space), derivatives will be matrices of a certain dimension, and directional/partial derivatives will be vectors in general, not just numbers. Also, it is possible for certain pathological (badly-behaved) functions to have all directional derivatives, but not be differentiable (i.e. not satisfy the limit definition).

$\endgroup$
  • $\begingroup$ You aren't rigorous as to what's a vector and what's a scalar. If h is a vector, then you should take the limit as it goes to the zero vector, not the limit as it goes to the zero scalar; both h and 0 should be bolded (as should x). Or you could phrase it as requiring that that expression in terms of of h*v goes to zero for all nonzero vectors v. If v is taken to be a unit vector, then |h v | is just h. This also simplifies the directional derivative to being with a specific v rather than over all v. $\endgroup$ – Acccumulation Jan 16 '18 at 17:01
  • $\begingroup$ They are vectors, I just didn’t know how to bolden them $\endgroup$ – Marcus Aurelius Jan 16 '18 at 20:12
  • $\begingroup$ You can search for "bolding". Here is one result: meta.mathoverflow.net/questions/2314/… $\endgroup$ – Acccumulation Jan 16 '18 at 20:15
  • $\begingroup$ You should write $f:\mathbb R^3 \to \mathbb R,$ not $f:\mathbb R^3\mapsto \mathbb R.$ the $\text{“}\mapsto\text{''}$ symbol is for things like $x\mapsto x^2 y^3$ or $y\mapsto x^2 y^3. \qquad$ $\endgroup$ – Michael Hardy Jun 7 '18 at 17:58
0
$\begingroup$

It is nothing more than the familiar chain rule! Let me elaborate.

We are in $\mathbb{R^n}$ and we want to take the diractional derivative to $u=(u_1,u_2,...,u_n)$ of the function $f:\mathbb{R^n}\rightarrow \mathbb{R}$ at some point $p_0=(p_1,...,p_n)$.

By definition $D_uf|_{p_{0}}=\lim_{t\rightarrow0}(\frac{f(p_0+tu)-f(p_0)}{t})$. By the chain rule $D(fog)|_p=Df|_{g(p)}Dg|_{p_0}$. So the limit becomes $<\nabla f|_{p_0},\frac{d(p_0+tu)}{dt}>= <\nabla f|_{p_0},u>$.

Note that with this method two vectors $a, l\cdot a$, will give different results even though they point to the same direction. Thats why some people allow only unitary vectors when the speak of directional derivative.

$\endgroup$
0
$\begingroup$

In basic calculus, you deal with functions that have one-dimensional input and one-dimensional output. When you take the derivative, you get another 1D -> 1D function. Once you start dealing with functions over more dimensions, however, things get much more complicated. The directional derivative is a way to simplify things back down. Instead of expressing a function f as being over vectors in general, i.e. f(v), you express the input as being some scalar h times a unit vector u. If u is fixed, you can have a function with a one-dimensional input by varying h; you can define gu(h) = f(h u). Now you can get a derivative g'(h) that is also just a simple, 1D -> 1D function.

Once you understand the concept of the directional derivatives, the next main idea that you need to understand is the idea that the derivative is linear. That is, if v = a+b, then g'v(h) = g'a(h) + g'b(h) (note that these vectors are not restricted to being unit). So if we have some basis i, j, k for our vector space, then we can find the derivative in any direction by decomposing v into the basis vectors, and multiplying the coefficients by the corresponding directional derivatives. So if v = c1i + c2j + c3k, then g'v = c1 g'i + c2 g'j + c3 g'k. Notice that that’s just the dot product of [c1,c2,c3] and [g'i,g'j,g'k]. The former is just the vector in the given basis, while the latter is the gradient in the given basis.

So now what is the direction of greatest ascent? For that, we want g'v/|v| to be maximum. We can use the Cauchy-Schwarz inequality to find that this happens when the vector is pointing in the same direction as the gradient. If we were to change to another basis [g, a, b] where g is the gradient, and a and b are orthogonal to g, then the directional derivatives of a and b will be zero, so the directional derivative of an arbitrary vector v will depend on the g component of v, which is maximum when v is in the same direction as g.

$\endgroup$
0
$\begingroup$

Let $s$ be a distance $u=\left<u_1,u_2,\ldots,u_n \right>$ a unit vector, $P(a_1,a_2,\ldots,a_n)$ a point and and $f(x_1,x_2,\ldots, x_n)$ be a real valued function differentiable in an open neighborhood $D$ of $P$

Then parameterize a line distance $s$ away from $P$ in the direction of $u$

$$ \begin{split} x_1=&a_1+su_1\\ x_2=&a_2+su_2\\ & \vdots \\ x_n=&a_n+su_n\\ \end{split} $$

Then apply the chain rule to find $\frac{df}{ds}$ along this line at least in the open neighborhood $D$ the following applies

$$ \begin{split} \frac{df}{ds}&=\frac{\partial f}{\partial x_1}\frac{d x_1}{d s} +\frac{\partial f}{\partial x_2}\frac{d x_2}{d s}+\cdots+ \frac{\partial f}{\partial x_n}\frac{d x_n}{d s}\\ &=\frac{\partial f}{\partial x_1}u_1 +\frac{\partial f}{\partial x_2}u_2+\cdots+ \frac{\partial f}{\partial x_n}u_n\\ &=\left<\frac{\partial f}{\partial x_1},\frac{\partial f}{\partial x_2},\ldots,\frac{\partial f}{\partial x_n} \right>\cdot \left<u_1,u_2,\ldots,u_n \right>\\ &= \nabla f \cdot u \end{split} $$

You can evaluate all the functions involved at $P$ to get $$ \frac{df}{ds} |_P= \nabla f(P) \cdot u = \left|\nabla f(P)\right| \cos \theta $$

where $\theta$ is the angle between $\nabla f$ and $u$. The biggest possible value for the directional derivative happens when $\cos \theta=1$ (this is the biggest possible value for cosine) which is when $\theta=0$. That is, $\nabla f$ and $u$ have to be in the same direction. Therefore, $\nabla f$ is in the direction of greatest increase of $f$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.