Gradient is a vector which we can obtain from any differentable function taking its partial derivatives.

From Wiki: "...the gradient points in the direction of the greatest rate of increase of the function"

I cant understand why this vector points in direction of greatest increase of a function! From definition of partial derivative it is a limit of ratio increment of function to increment of argument. Increment of argument, by definition, should tend to zero. So argument should be positive? But for some function increasing the argument we can receive lesser value of function

For example lets consider function

y = -x^2

-2x or (1, -2x)?

How I can find direction of increase of this function in point (-1;-1)?

Feeling lost. Sorry for mess.

Let's look at your one dimensional case. The gradient of the function $f\left(x\right) = -x^2$ is the one-dimensional vector $-2x$, which is itself a function of x. Say x is positive, then the gradient is negative, because the function $-x^2$ does indeed increase as x gets less positive. If x is negative, the gradient is positive, because the function does indeed increase as x gets less negative (it's maxiumum is at 0). The key here is that the gradient is itself a function of x, and it will be negative on the regions where the function is decreasing (since it will be increasing in the other direction).

The same goes for multiple dimensions. Consider, for example:

$f\left(x,y\right)=x^2 - y^3$

then the gradient of f would be the vector

$\left(\begin{array}{c} \frac{\partial z}{\partial x}\\\frac{\partial z}{\partial y}\end{array}\right)=\left(\begin{array}{c} 2x\\-3y^2\end{array}\right)$

Which would indeed still point in the xy-direction of steepest ascent.

• So the sign of component of gradient vector tells us in which direction to move - increase or decrease the argument, right? $f(x) = -x^2$, $grad(f(x)) = -2x.$ If argument $x$ is positive then gradient points in negative direction along axis $x$ which tell us to decrease the argument to get bigger value of a function? – Il'ya Zhenin Dec 11 '14 at 10:15
• That is correct. – SBareS Dec 11 '14 at 16:20
• Thank you,I think I got it :) – Il'ya Zhenin Dec 12 '14 at 10:33

Let's try to prove, that the gradient indeed points in the direction of the steepest rate of increase at a point $x_0$.

Assume $f(x)\in\mathbb{R}$ is differentiable function at $x_0$, where $x \in \mathbb{R}^n$. Now assume, that we want to compute the rate of increase with respect to a particular direction $v \in \mathbb{R}^n$ at $x_0$, where $\left|v\right| = 1$ (The vectors points in the direction we are trying to compute the rate of increase and has length one, because we only care about the direction and not the length of the vector).

Now, the values the function is taking along that direction $v$ around $x_0$ can be described by the function $f(vt + x_0)$, for $t$ in some interval $(-\epsilon,\epsilon), \epsilon > 0$. Now in order to find out the rate of increase $r_v$ at $x_0$ along the positive direction of $v$, we just have to differentiate $f(vt+x_0)$ at $t=0$, thus $r_v=\frac{d}{dt} f(vt+x_0)|_{t=0}$.

Computing $r_v$ gives us: $r_v = \nabla f^T(x_0)v$ (Using chain-rule and convention that $\nabla f$ is a column vector).

Ok, now we have computed the generic rate of increase $r_v$ at $x_0$, with respect to every direction $v$ in $\mathrm{R}^n$. The original question was now, to find the direction $v$, such that $r_v$ is maximized.

Now, notice that the maximum value for $r_v$ will be obtained exactly then, if $v$ points in the same direction as $\nabla f(x_0)$, thus $v_{opt} = \frac{\nabla f(x_0)}{\left|\nabla f(x_0)\right|}$. Hence, $\nabla f(x_0)$ is indeed pointing always in the direction of the steepest increase.

Mathematicall speaking the optimum value for v follows from: $r_v = \nabla f^T(x_0)v \leq |\nabla f(x_0)||v| = |\nabla f(x_0)|$, since $|v|=1$ and this upperbound can be achieved by $v_{opt} = \frac{\nabla f(x_0)}{\left|\nabla f(x_0)\right|}$.

As a person focused on applications, I prefer intuitive proofs to rigorous proofs.

Let’s work with 2 dimension function $\color{blue}{f(x,y)}$. (the reasoning is easily extendable to more dimensions)

$\color{blue}{ \nabla f = [ \delta f / \delta x , \delta f / \delta y ]}$

We want to estimate a function value near a point $\color{blue}{P(x_1, x_2)}$, with $\color{blue}{\Delta x}$ variation in X axis and $\color{blue}{\Delta y}$ in Y axis.

Well, if we measure the function variation rate in X axis ($\color{blue}{\delta f / \delta x}$ ) and function variation rate in Y axis ($\color{blue}{\delta f / \delta y}$) it is easy to accept that

$\color{blue}{f(near P) = f(P) + (\delta f / \delta x ) * \Delta x + (\delta f / \delta y ) * \Delta y}$ (1)

This intuitive claim can be proved with chain rule in Calculus.

It’s can be written like this:

$\color{blue}{f(near P) = f(P) + \nabla f .[ \Delta x, \Delta y ]}$

Following the dot product definition.

Now imagine a level curve$\color{blue}{ f(x,y) = c}$, where $\color{blue}{c}$ is a constant. If $\color{blue}{\Delta x}$ and $\color{blue}{\Delta y}$ tends to $\color{blue}{0}$ along this level curve, it’s obvious that $\color{blue}{f(near P) = c}$ , i.e., the dot product is $\color{blue}{0}$.

The alternative and equivalent definition for dot product is

$\color{blue}{u.v = \|u\|\|v\|.cos \theta}$ (2)

Much explanation ends up leading to the need for further explanation. Why algebric dot product definition implies in geometric dot product definition? Well, here is an proof of the equivalence of the algebraic definition of dot product with geometric definition and here is the proof of equivalence of geometric definition of dot product (best starting point for an intuitive approach to the definition) with algebraic definition.

$\color{blue}{u * v = \|u\|\|v\| cos \theta}$

In geometric definition, if $\color{blue}{u}$ orthogonal to $\color{blue}{v}$, $\color{blue}{\theta = \pi /2}$ , so $\color{blue}{u.v = }0$.

So gradient of f function should be perpendicular to level curve, because $\color{blue}{[ \Delta x, \Delta y]}$ is tangent to level curve in P for smalls $\color{blue}{\Delta x}$ and $\color{blue}{\Delta y}$

Finally, why gradient is the steepest function variation?

If one look at (1) and (2), one can conclude that the bigger variation of $\color{blue}{f(near P)}$ happens when $\color{blue}{\theta = 0}$, i.e, $\color{blue}{[ \delta f / \delta x , \delta f / \delta y]}$ and $\color{blue}{[\Delta x, \Delta y]}$ are parallel. Therefore $\color{blue}{[\delta f / \delta x , \delta f / \delta y]}$ points at the greater variation of $\color{blue}{f(near P)}$

Besides, an orthogonal direction is the fastest way to get away from a level curve and, therefore, to approach the next level curve! It’s easy to see this in a map with level curve linking points of same height.

♦♦♦

Regarding your example function $\color{blue}{y = -x^2}$ has gradient $\color{blue}{-2x}$

$\color{blue}{y}$ is a some value (height, color, power, etc.) related to $\color{blue}{x}$. In an unidimensional world the gradient does not make much sense because there is only one possible direction with 2 orientations:

When $\color{blue}{x>0}$, gradient is negative, so $\color{blue}{y}$ decrease for growing $\color{blue}{x}$. When $\color{blue}{x<0}$, gradient is positive, so $\color{blue}{y}$ decrease for decreasing $\color{blue}{x}$.