12
$\begingroup$

I understand that differentiation of a function ($\mathbb{R} \rightarrow \mathbb{R} $) at a point is the rate of change in the output for a slight nudge in the input. And this rate of change could be negative or positive. There is no concept of direction for the single-variable function as obvious.

Now, my doubt is in the case of the multivariate function ($\mathbb{R}^n \rightarrow \mathbb{R}$) where differentiation is a gradient. And this gradient representing partial differentiation w.r.t. to each basis becomes a direction. This direction is a direction of ascent but not descent, why?. Why it is a direction is of ascent. My question is not at all related to steepest ascent, about which one can find many answers on this forum and read elaborately at this link. An intuitive explanation would be preferable than mathematical at this link.

$\endgroup$
  • $\begingroup$ What does "direction of ascent [ascendent, I suppose]" mean in $\;\Bbb R^n\;$ ? $\endgroup$ – DonAntonio Mar 16 '18 at 13:34
  • 2
    $\begingroup$ The same interpretation also works for $\Bbb R\to\Bbb R$! There is a concept of direction for single-valued functions: left and right. Your derivative is a 1-dimensional vector (a single number). When the number is positive, it is a vector pointing in the positive direction of the number line (to the right). But since the derivative is positive, this is also the direction of ascent. $\endgroup$ – M. Winter Mar 16 '18 at 13:41
  • 3
    $\begingroup$ Gradient is just the matrix of the differential. For a real function, this looks like a vector and the differential is the dot product of its matrix times the coordinate of the increment of the variable. The dot product is largest when the vectors point in the same direction (Cauchy-Schwartz inequality). The differential is defined as the best linear approximation of the increment $f(x+h)-f(x)$. Therefore, in a small enough neighborhood the largest increment $f(x+h)-f(x)$ is close to the largest value of the differential (when $h$ and the gradient are pointing in the same direction). $\endgroup$ – YAlexandrov Mar 16 '18 at 13:57
  • 2
    $\begingroup$ To continue what @YAlexandrov is saying, it's basically because we choose $f(x+h)-f(x)$ in the definition of gradient rather than $f(x)-f(x+h)$ $\endgroup$ – Tony S.F. Mar 16 '18 at 14:04
19
$\begingroup$

The comments persuaded me to reformulate my answer. For the original (still correct, but sub-optimal) version, see below.

The gradient is defined in a completely natural way. There is no completely mathematical reason, why it can be said to point to the steepest ascent. It has more to do with some more or less arbitrary choices being made in several definitions, which break this symmetry.

Observation. The concept "gradient points in direction of ascent" also works for single-valued functions $\Bbb R\to\Bbb R$. There is indeed a concept of direction in $\Bbb R$: right and left. A positive derivative is a vector (the gradient) pointing to the right (in the direction of ascent), a negative derivative is a vector pointing to the left (in this case, also the direction of ascent, because the function is decreasing to the right). So since the same observation also applies in 1D, we should start looking for an explanation here.

Note: I am going to use the terms "right" and "left" for the direction "positive" and "negative" on the number-line, because this is the standard orientation of the number-line. This is also a symmetry break, but only a notational one. It does not effect the mathematics in any way if we flip these directions.

Thanks to the comments, some few definitions could be localized as the rootcause of the broken symmetry. If you are standing on a mountain side, there is no meaning in asking whether it is up-hill or down-hill. This question only makes sense if you define a direction with respect to which we should judge the slope. The same goes for single-valued functions. It has been standard to call a function increasing if it's function graph is uphill to the right. This involves two arbitrary choices:

  1. The $y$-axis is pointing upwards, hence increasing function values are seen as going up. This is the most obvious arbitrary choice. Many applications do it the other way around, e.g. line-numbers in text are increasing from top to bottom, and pixels on a screen are usually adressed with an downwards-increasing $y$-axis.
  2. The kind of slope is judged w.r.t. to the "arbitrary" direction "right". Why not left? It seems natural, but is not forcing.

There might be another arbitrary choice: a positive derivative indicates that the function is increasing. We could have defined it the other way around. Anyways, flipping any single of these definitions will change the gradient from pointing upwards to pointing downwards.

Note. Yes I know, "increasing" is formally defined as $x\le y\implies f(x)\le f(y)$, but also this definition is motivated by the visualization of an increasing function graph to the right. No one would use it if the $y$-axis was pointing downwards.

Conclusion: The reason for the gradient pointing to the steepest ascent is based in our somewhat biased definitions. This is especially evident in the 1D-case. The derivative is defined in such a way so that it has a positive value (the gradient points to the right) if the function increases. A function is called increasing if its function graph is going uphill to the right. You see how these arbitrary definitions combine to "gradient is pointing uphill".


ORIGINAL

Because we have a somehow biased definition of differentiation. Let me explain.

As noted in a comment, this "gradient pointing in direction of ascent" also works for single-valued functions $\Bbb R\to\Bbb R$. There is indeed a concept of direction in $\Bbb R$: left and right. A positive derivative is a vector (the gradient) pointing to the right (in the direction of ascent), a negative derivative is a vector pointing to the left (also in the direction of ascent, because the function is decreasing). So since the same observation happens in $1$D, we should probably start there looking for an answer.

It all happens because the definition of derivative is biased in some sense. Someone once decided that a function is considered increasing if its value gets bigger to the right. You see the broken symmetry? Why to the right, why not to the left? So once one decided that the derivative is positive $-$ the gradient points to the right $-$ when the function grows to the right. Here we have it. Whoever defined it, directly coupled the terms "direction of gradient" and "direction of acsent".

Would he had decided to define a function as increasing if its value grows to the left (unnatural, considering our left-to-right reading direction), then the gradient would point to the steepest decent.

Note: In this answer I assumed that the number line is oriented with the positive number on the right. This is standard, but another symmetry break (but only a notational one, without impact on the mathematics). You can substitute all left/right above by negative/positive if you want to be indepedent of this broken symmetry.

$\endgroup$
  • 2
    $\begingroup$ I think you're missing the root cause -- that we put the positive numbers on the right side of the number line and the negative numbers on the left side. $\endgroup$ – Hurkyl Mar 16 '18 at 14:16
  • $\begingroup$ @Hurkyl Why I am missing that? I assume this is known. I could have written "positive" instead of "right" and "negative" instead of "left". But it helped my point. $\endgroup$ – M. Winter Mar 16 '18 at 14:17
  • 11
    $\begingroup$ E.g. what was actually decided is that a function is considered increasing if $x \leq y$ implies $f(x) \leq f(y)$. This has nothing to do with right or left or any other sort of direction. Getting bigger to the right is simply because that's the direction we drew the number line. $\endgroup$ – Hurkyl Mar 16 '18 at 14:19
  • 3
    $\begingroup$ @Hurkyl You are right, the direction of the number line is also a symmetry break, but a purely notational one. We could flip the number line and no one would notice as long as all our definitions never refer to left/right, but only to negative/positive. $\endgroup$ – M. Winter Mar 16 '18 at 14:27
  • 6
    $\begingroup$ I think that this answer misses an important point: Any "bias" in the derivative is due to a pairing between linear functions and vectors - which is natural given inner product structure. In particular, $\nabla f(x)$ is defined so that $\nabla f(x)\cdot \Delta$ is the best linear approximation (in $\Delta$) to $f(x+\Delta)-f(x)$. This implies that $\nabla f(x)$ points to higher values of $f$. (And without inner product structure, one just takes the derivative to be a linear function, which is in the dual vector space, avoiding this issue entirely) $\endgroup$ – Milo Brandt Mar 16 '18 at 19:42
4
$\begingroup$

"There is no concept of direction for the single-variable function as obvious."

False.

When the univariate derivative is positive, the function increases in the direction of increasing input; when negative, in the direction of decreasing input. There are only two possible directions, but whichever it is, the derivative does point in the direction in which the function increases.

$\endgroup$
4
$\begingroup$

The gradient with regard to some input variable indicates how much the value of the output variable goes up (i.e. ascends) when that input variable goes up. As such, if you move in the direction of the gradient, and the gradient is positive, then the value of the output variable will go up. If the gradient is negative, however, the increasing the input variable will decrease the output variable. But yes, a positive gradient means that you will ascend if you 'follow' the gradient.

This is our definition of gradients or slopes, and works just the same in $\mathbb{R}$ as in $\mathbb{R}^n$. That is, when we take the derivative, the derivative $\frac{dy}{dx}$ indicates to what extent $y$ increases as $x$ increases which, by the way, is the same as the extent to which $y$ decreases as $x$ decreases.

Now, some answers suggest that when we defined the gradient in this manner there was a bit of 'arbitrariness' involved, and that we could have defined the gradient differently so that the direction would reverse. It is even suggested that this has something to do with 'right' being arbitrarily chosen as the 'up' direction and 'left' being 'down'.

However, I strongly disagree with those answers, because the alternative would have been to say that the gradient would be the extent to which what extent $y$ increases as $x$ decreases .. which would be a very confusing and unnatural thing to do; it's like bringing in an extraneous negative or reversal.

Anyway, when you want the output value to descend, you should go in the 'other' direction of the gradient, i.e. subtract a value proportional to that gradient. Of course, that does mean that if the gradient is negative, you end up increasing the input variable in order to decrease the output variable.

$\endgroup$
  • $\begingroup$ Are you sure? When you look at the multivariate function from the prism of a single variable (keeping all the other variables constant) then function value might go up or down (when the partial derivative is positive up and negative down) even though input variable goes up. For example in function $f(x, y) = x^2 - 2y$, keep y fixed at 2 then new function just variable of $x$ is $x^2 - 4$ and its differentiation $2x$ i.e. -2 at $x = -1$. Function value at $f(-1, 2)$ is -3. Now move slightly positive in the direction of x to -0.98 then $f(-0.98, 2)$ is -3.0396 which is smaller than -3. $\endgroup$ – zeal Mar 16 '18 at 14:48
  • $\begingroup$ @zeal Right, that's entirely consistent with what I said. You want to subtract proportional to the gradient in order to get a smaller value. Since the gradient in this case is negative, you thus subtract a negative number, i.e. make the variable go up a little bit ... and as you show, the function value does indeed go down when you do so. $\endgroup$ – Bram28 Mar 16 '18 at 16:43
3
$\begingroup$

A more general definition of "derivative" than what you are used to would be "linear function $L(a) = f'(x)$ that best approximates $f(a+x)$ with small $a$".

For a function $\mathbb{R}\rightarrow\mathbb{R}$ this linear function is also from $\mathbb{R}\rightarrow\mathbb{R}$.

We can identify the linear functions $\mathbb{R}\rightarrow\mathbb{R}$ with a point in $\mathbb{R}$ by noting $L(a) = ka$ for some constant $k$.

We could, if we chose, define $L(a) = -ka$, or $L(a) = \pi k a$ or $L(a) = ae^k$ just as easily and get a different $k$. The easy one we pick ends up making some of the math easier, plus we get addition of derivatives corresponding to addition of the scalar we associate with them and even multiplication working out.

When we take this definition and extend it to $\mathbb{R}^n \rightarrow \mathbb{R}$ or $\mathbb{R} \rightarrow \mathbb{R}^m$ or even $\mathbb{R}^n \rightarrow \mathbb{R}^m$ we still get derivatives being linaer functions, but now between different spaces.

These linear functions can be mapped to matrices, and in some cases vectors. If this mapping is linear, this mapping is dependent on your choice of basis.

Working with vectors and matrices is often easier than working with an abstract derivative. So we pick a basis that makes the rest of the math work better.

Given $f: \mathbb{R}^n \rightarrow \mathbb{R}$, we get $L(a) = f'(x)$ as the best linear approximation to $f(a+x)$, aka

$$f(a+x) \approx L(a) + f(x)$$

If $L$ is represented as a vector $G$ such that $L(a) = G \cdot a$, then along the line $L(a) \leq |G||a|$ and $L(a) = \lambda |G|^2$ if $a=\lambda G$.

In short, the derivative is maximal along the line parallel to the direction pointed by $G$, the gradient vector, which is our chosen representiative for the linear function that is the derivative of $f$ at $x$.

$\endgroup$
2
$\begingroup$

It's all a matter of definition. For whatever reason, the first people to formalize all this decided that the direction of ascent is the thing that needed a name and the direction of descent did not. It all works out either way, though. If the term "gradient" had been slapped on the direction of descent, we would just have to put minus signs in front of many accepted results; that's all.

$\endgroup$
1
$\begingroup$

To first order, the increase of $f$ at $a$ in the direction $u$ is given by $f(a + hu) - f(a) = h D_u f(a)$. To find the direction $u$ which maximizes this increase, we look at the directional derivative of $f$ at $a$ in the direction $u$, $$ D_u f(a) = \lim_{\varepsilon \to 0} \frac{f(a + \varepsilon u) - f(a)}{\varepsilon} = u \cdot \nabla f(a), $$ and observe that picking $u$ to point in the direction of $\nabla f(a)$ gives the largest value, i.e., the gradient is the direction of maximal ascent. If we instead wanted the direction of maximal descent, we'd start with $f(a) - f(a+hu) = - h D_u f(a)$ and in the same way see that the negative gradient provides the direction of maximal descent.

All of the above also holds on one dimension, and hence one could argue that there is a concept of direction in one dimension. Intuitively, it's because the derivative $f'(a)$ measures the increase of $f$ at $a$ in the positive direction. More concretely, note that there are just two unit vectors, namely $+1$ and $-1$, in one dimension. The direction of maximal increase is still found by picking the unit vector $u$ to maximize $u \cdot \nabla f(a)$, which is better written in one dimension as $u f'(a)$. If the derivative $f'(a)$ is positive, the direction is $u = +1$ (or "right"), and if the derivative is negative, the direction is $u = -1$ (or "left"); in either case, the derivative gives the direction of maximal increase (through its sign).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.