30
$\begingroup$

I'm starting to learn how to intuitively interpret the directional derivative, and I can't understand why you wouldn't scale down your direction vector $\vec{v}$ to be a unit vector.

Currently, my intuition is the idea of slicing the 3D graph of the function along its direction vector and then computing the slope of the curve created by the intersection of the plane.

But I can't really understand how the directional derivative would be a directional derivative if it were not scaled down to be a change in unit length in the direction of $\vec{v}$. Is there an intuitive understanding I can grasp onto? I'm just starting out so maybe I haven't gotten there yet.

Note, I think there may be a nice analogy to linearization, like if you take "twice as big of a step" in the direction of $\vec{v}$ , then the change to the function due to the change in this step is twice as big. Is this an okay way to think about it?

$\endgroup$
  • 19
    $\begingroup$ Why the downvote for such a question? It is well-asked, not too broad and the author shares its current insight. It is an interesting question that deserves all the upvotes! $\endgroup$ – C. Falcon May 22 '17 at 2:12
  • 1
    $\begingroup$ Thank you so much @C.Falcon! $\endgroup$ – rb612 May 22 '17 at 2:24
  • 1
    $\begingroup$ Silly question: wouldn't you end up with a notion that intrinsically depends on the particular norm you choose? For instance, you would have one directional derivative defined for Euclidean, one for L1... $\endgroup$ – Clement C. May 22 '17 at 9:58
  • 4
    $\begingroup$ Why is the usual derivative of a function on the real line not scaled down? $\endgroup$ – Thompson May 22 '17 at 18:18
  • 1
    $\begingroup$ @MartianInvader We can simply exclude zero. $\endgroup$ – Alex Vong May 23 '17 at 12:05
19
$\begingroup$

The intuition I think of for a directional derivative in the direction on $\overrightarrow{v}$ is that it is how fast the function changes if the input changes with a velocity of $\overrightarrow{v}$. So if you move the input across the domain twice as fast, the function changes twice as fast.

More precisely, this corresponds to the following process that relates calculus in multiple variables to calculus in a single variable. In particular, we can define a line based at a point $\overrightarrow{p}$ with velocity $\overrightarrow{v}$ parametrically as a curve: $$\gamma(t)=\overrightarrow{p}+t\overrightarrow{v}.$$ This is a map from $\mathbb R$ to $\mathbb R^n$. However, if $f:\mathbb R^n\rightarrow \mathbb R$ is another map, we can define the composite $$(f\circ \gamma)(t)=f(\gamma(t))$$ and observe that this is a map $\mathbb R\rightarrow\mathbb R$ so we can study its derivative! In particular, we define the directional derivative of $f$ at $\overrightarrow{p}$ in the direction of $\overrightarrow{v}$ to be the derivative of $f\circ\gamma$ at $0$.

However, when we do this, we only see a "slice" of the domain of $f$ - in particular, we only see the line passing through $\overrightarrow{p}$ in the direction of $\overrightarrow{v}$. This corresponds to the notion of slicing you bring up in your question. In particular, we do not see any values of $f$ outside of the image of $\gamma$, so we are only studying $f$ on some restricted set.

$\endgroup$
  • $\begingroup$ Thank you! Can you elaborate more on hitting a slice? As regardless of magnitude, if you are moving with the same direction, you will stay on the same plane whether it's unit vector $\vec{v}$, or $n*\vec{v}$ $\endgroup$ – rb612 May 22 '17 at 2:03
  • 6
    $\begingroup$ Some authors actually require $\vec{v}$ to be a unit vector (Stewart). However, I prefer to allow $\vec{v}$ to be arbitrary length. $\endgroup$ – Michael Burr May 22 '17 at 2:06
  • $\begingroup$ @rb612 I added a bit more rigor to the comment I made on slicing - does that clarify? $\endgroup$ – Milo Brandt May 22 '17 at 2:11
  • $\begingroup$ @MichaelBurr Huh. I didn't know that. I'm not sure how to address that in an answer to a question that is certainly asking about a definition that does not require that, though. (I wonder why they do that. I can think of lots of good uses non-unit vector directional derivatives) $\endgroup$ – Milo Brandt May 22 '17 at 2:21
  • $\begingroup$ I was just adding to the conversation, there's nothing to address, the use of only unit vectors bothers me too. $\endgroup$ – Michael Burr May 22 '17 at 2:38
15
$\begingroup$

Let $f : \mathbb{R}^n \to \mathbb{R}^m$ and (if the limit exists) $$D_v f(x) = \lim_{h \to 0} \frac{f(x+hv)-f(x)}{h}$$ be the directional derivative in the direction $v$. This way, if the function is differentiabble $$ D_{au+bv} f(x) = a\, D_{u} f(x)+b\, D_{v} f(x) \qquad (a,b) \in \mathbb{R}^2$$ ie. the directional derivative is linear in the direction. Indeed $$D_{v} f(x) = J_x v$$ where $J_x$ is the Jacobian matrix.

You'll have some problems for saying and understanding that if you restrict to $\|v\|=1$, or worse if you normalize $D_vf(x)$

$\endgroup$
  • 1
    $\begingroup$ Indeed, the linearity of the directional derivative, or Jacobian, is the crucial property which makes it so useful. $\endgroup$ – Robin Saunders May 22 '17 at 3:49
12
$\begingroup$

Unit vectors are vastly overrated — the notion of vector is far more computationally convenient when treated as a whole rather than decomposed into separate notions of direction and magnitude.

I claim it leads to better understanding as well.

Thus, one should not introduce unit vectors by habit — such a manipulation should be reserved for those circumstances when it does something useful.

Similarly, a good definition or computational tool shouldn't force unit vectors upon the user, unless there is a very good reason for doing so.


Algebraically, the directional derivative is not the main idea — the main idea is the differential of a function: in usual terms, $\nabla f$ is the row vector given by

$$ \nabla f(\vec{x}) = \begin{pmatrix} f_1(\vec{x}) & f_2(\vec{x}) & f_3(\vec{x}) \end{pmatrix} $$

where by $f_k$, I mean the derivative of the function $f$ in its $k$-th place. The directional derivative is simply the product of the differential with the given (column) vector:

$$ \nabla_\vec{v} f = (\nabla f) \vec{v} $$

As such, restricting to unit vectors is unnatural thing to do. Rescaling the input vector to be a unit vector is extremely unnatural.

Note that some people use $\nabla f$ to refer to a column vector, or even treat row and column vectors as the same thing. This is unfortunate, because it is computationally awkward when you change variables, and gets in the way of understanding the difference between vectors and covectors, and the close relationship between the inner product and the transpose operation.


Finally, it's worth noting that derivatives — even directional derivatives — make sense in contexts where there is no notion of length, and thus there is no notion of a "unit" vector that can be applied.

$\endgroup$
  • 1
    $\begingroup$ Good answer. By the way, one point in favor of having $\nabla f(x)$ be a column vector is that in optimization when using gradient descent we want to move in the direction of steepest descent, so we have formulas like $x^{k+1} = x^k - t \nabla f(x^k)$. If $\nabla f(x^k)$ weren't a column vector, it would be somewhat awkward to describe this. $\endgroup$ – littleO May 22 '17 at 7:13
  • 1
    $\begingroup$ Right. But gradient descent only works in situations where there is an inner product, which then induces a natural correspondence between covectors and vectors. So you could equally write $I(\nabla_x f)$ instead of just $\nabla_x f$ (or whatever notation you prefer). $\endgroup$ – Robin Saunders May 22 '17 at 15:35
  • $\begingroup$ I'd agree that this is probably the best answer. $\endgroup$ – Robin Saunders May 22 '17 at 15:36
  • $\begingroup$ @RobinSaunders: Right but that only affirms littleO's point: the vector vs. covector distinction frequently buys you nothing but notational inconvenience, so why bother. $\endgroup$ – Mehrdad May 23 '17 at 5:53
  • 1
    $\begingroup$ The reason to bother is that an understanding of the difference between vectors and covectors provides valuable insight even when there is a natural correspondence between them, and is necessary in the many situations where there isn't. As for notational inconvenience, it's completely standard to drop notational baggage where it can be inferred, and this is not unique to the vector/covector distinction, but understanding that something is being dropped in the first place is still worthwhile (as with, for example, dropping arguments from a function when they're implied). $\endgroup$ – Robin Saunders May 23 '17 at 17:51
8
$\begingroup$

I used to feel uncomfortable about this also. One point is that there is no harm in allowing $\vec v$ not to be a unit vector, and it is arguably simpler to omit this requirement because it's not necessary anyway. Another point is that it is sometimes interesting and useful to think of the directional derivative $D_{\vec v}f(x)$ as a function of $\vec v$, with $x$ held fixed. This function has the nice property that if you scale the input, the output gets scaled the same way. But in order to make this statement, we must not require $\vec v$ to be a unit vector.

$\endgroup$
3
$\begingroup$

I originally left comments on other answers, but perhaps they deserve to be combined into an answer of their own.


To make the reasoning in Milo's answer less abstract, imagine the function $f$ that we're interested in gives the altitude at a given point of land, and we're driving around. Then our velocity as we pass through the point $p$ is given by some vector $v$, and we can work out how fast our altitude is changing by finding the directional derivative of $f$ in the direction of $v$ (at point $p$).

You should really think of directional derivatives in terms of a function $\nabla_p$, the gradient of $f$ (at $p$), which takes any vector based at $p$ as input and gives the directional derivative of $f$ in the direction of $v$ (at point $p$) as output. As a function of vectors based at $p$, $\nabla_p$ is linear (as user1952009 indicated), and this is what makes it so useful: for example, it follows that for any two vectors $v$, $w$, $\nabla_p(v+w) = \nabla_p(v) + \nabla_p(w)$. And, as you noted, $\nabla_p(av) = a\nabla_p(v)$ for any scalar $a$.

In general, the reason derivatives are useful in the first place is precisely because they allow us to approximate arbitrary differentiable functions near a given point using only linear functions. The latter are far simpler, with the nice behaviour illustrated above, which enables many useful constructions - first in single- and multi-variable calculus, and later in differential and Riemannian geometry. For example, the fundamental theorem of calculus (that differentiation and integration are "inverse" operations) generalizes to Stokes' theorem for manifolds, a result which is both beautiful and used in an incredibly diverse range of settings.

$\endgroup$
2
$\begingroup$

Two such reasons:

  1. This will fail for the zero vector.

  2. Do you normalize derivatives for the 1-dimensional case? Should you? Most would say no.

$\endgroup$
  • $\begingroup$ Actually the raw derivative in single argument calculus is "normalized". $D_{\Delta x}f\left[x_{0}\right]=\lim_{t\to0}\frac{f[x_{0}+t\Delta x]-f[x_{0}]}{t}$ $=\Delta x\lim_{t\to0}\frac{f[x_{0}+t\Delta x]-f[x_{0}]}{t\Delta x} =\Delta xf^{\prime}\left[x_{0}\right].$ If $\Delta x=1,$ then $D_{\Delta x}f\left[x_{0}\right]=f^{\prime}\left[x_{0}\right].$ $\endgroup$ – Steven Thomas Hatton Dec 9 '17 at 16:47
1
$\begingroup$

Here's a decent reason to have the thing we call the directional derivative. And not require the reference vector to be of unit magnitude.

Start with the case of $f:\mathbb{R}\to\mathbb{R}$. Write the directional derivative with respect to some arbitrary constant $h\ne0$, and apply first principles

$D_{h}f\left[a\right]=\lim_{t\to0}\frac{f[a+th]-f[a]}{t}$

$=h\lim_{t\to0}\frac{f[a+th]-f[a]}{th}$

$=hf^{\prime}\left[a\right]$.

Neither surprising nor particularly interesting. It's just another way of saying what we already knew.

$x\left[t\right]=a+th$

$g\left[t\right]=f\left[x\left[t\right]\right]$

$g^{\prime}\left[0\right]=f^{\prime}\left[a\right]x^{\prime}=hf^{\prime}\left[a\right].$

But it gets a bit more interesting if we use it to build up a Taylor series for a real-valued function of a real argument. I'll leave the notation to speak for itself.

$D_{h}D_{h}f\left[a\right]=D_{h}^{2}f\left[a\right]=h^{2}f^{\prime\prime}\left[a\right]$.

$D_{h}^{r}f\left[a\right]=h^{r}f^{(r)}\left[a\right]$.

$D_{h}^{0}f\left[a\right]=f\left[a\right]$.

$f(a+h)=\sum_{r=0}^{k}\frac{D_{h}^{r}f\left[a\right]}{r!}+\frac{D_{h}^{k+1}f\left[\zeta\right]}{(k+1)!}$.

I think that's kind of pretty.

Now lets call $f:\mathbb{R}^{n}\to\mathbb{R}$. I use this non-standard notation as a shorthand:

$\{\mathbb{J}\}=\left\{ \left\{ j_{i}\geq0\right\} _{n}|\sum_{i=1}^{n}j_{i}=k\right\} $.

Think of the $\mathbb{J}^{'s}$ as multi-indices. Each $\mathbb{J}$ is a set of non-negative integers $\{j_{1},\dots,j_{n}\}$ such that

$j_{1}+\dots+j_{n}=k$.

Now use those indecies to write the multinomial expansion of a sum of n terms:

$\left(x_{1}+\dots+x_{n}\right)^{k}=\sum_{\mathbb{J}}\begin{pmatrix}k\\ j_{1}\dots j_{n} \end{pmatrix}x_{1}^{j_{1}}\dots x_{n}^{j_{n}}$.

Here the coefficients are the multinomial generalization of the binomial coefficients:

$\begin{pmatrix}k\\ j_{1}\dots j_{n} \end{pmatrix}=\frac{k!}{j_{1}!\dots j_{n}!}=\begin{pmatrix}k\\ \mathbb{J} \end{pmatrix}$.

Now write the Taylor series as before, but using $\mathfrak{h}\in\mathbb{R}^{n}$.

$f(\mathfrak{a}+\mathfrak{h})=\sum_{r=0}^{k}\frac{D_{\mathfrak{h}}^{r}f\left[\mathfrak{a}\right]}{r!}+\frac{D_{\mathfrak{h}}^{k+1}f\left[\vec{\zeta}\right]}{(k+1)!}$

$=P_{k}\left[\mathfrak{h}\right]+R_{k}\left[\mathfrak{h}\right].$

The directional derivative is:

$D_{\mathfrak{h}}f\left[\mathfrak{a}\right]=\mathfrak{h}\cdot\nabla\left[f\left[\mathfrak{a}\right]\right]=\left(h_{1}D_{1}+\dots+h_{n}D_{n}\right)\left[f\left[\mathfrak{a}\right]\right].$

So:

$D_{\mathfrak{h}}^{r}f\left[\mathfrak{a}\right]=\left(h_{1}D_{1}+\dots+h_{n}D_{n}\right)^{r}\left[f\left[\mathfrak{a}\right]\right].$

So the Taylor polynomial of degree $k$ is:

$P_{k}\left[\mathfrak{h}\right]=\sum_{r=0}^{k}\sum_{\mathbb{J}}\begin{pmatrix}r\\ j_{1}\dots j_{n} \end{pmatrix}\left(h_{1}D_{1}\right)^{j_{1}}\dots\left(h_{n}D_{n}\right)^{j_{n}}\left[f\right]$.

The $\left[f\right]$ indicates that the beast to the left is an opperator being applied to $f$.

$\endgroup$
  • $\begingroup$ If course all of that can be done with unit vectors and real numbers with which to scale the unit vectors. Shall we call such real numbers "scalars"? Actually, it is probably a productive exercise to differentiate only with respect to unit vectors, and adjust things later. $\endgroup$ – Steven Thomas Hatton Dec 8 '17 at 3:11
  • 1
    $\begingroup$ In $f(a+h)=\sum_{r=0}^{k}\frac{D_{h}^{r}f\left[a\right]}{r!}+\frac{D_{h}^{k+1}f\left[\zeta\right]}{(k+1)!}$ you should write $ \zeta_a(h)$ and say it is in $[0,h]$ (the Taylor remainder) $\endgroup$ – reuns Dec 8 '17 at 18:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.