I'm having trouble understanding why we can obtain direction of maximum increase(gradient) from simply using the partial derivatives as components of the vector? What reasoning is behind this? How it was discovered / proven that standing on inclined plane, the relation between the directional derivative in angle x and the angle x is a sinusoid (i gather this must have been known to the man who started to use dot product to calculate directional derivative from gradient)? Thanks in advance

Edit: to simplify - Why partial derivations of scalar-valued function put into vector give direction of maximum increase?

  • $\begingroup$ Think about why this works when $f(x,y)$ is a function of $x$ or $y$ only. $\endgroup$ – Jim Belk Jun 21 '13 at 16:27

The big idea here is this: assuming that $f$ is "nice enough", the directional derivative $D_{\vec{v}}f(\vec{x})$ in the direction $\vec{v}$ (which is a unit vector) can be expressed by $$ D_{\vec{v}}f(\vec{x})=\vec{v}\cdot\nabla f(\vec{x}), $$ where $\nabla f=\langle\frac{\partial}{\partial x_1}f,\ldots,\frac{\partial}{\partial x_n}f\rangle$ is the gradient of $f$. Why? The rough reason is that if $f$ is differentiable, then $$ f(\vec{x}+\Delta\vec{x})\approx f(\vec{x})+\nabla f(\vec{x})\cdot\Delta\vec{x}. $$ (This is the multivariate generalization of the linear approximation, and can be though of as following from using the linear approximation in each coordinate.) Then by definition, for $h$ small the directional derivative is $$ D_{\vec{v}}f(\vec{x})\approx\frac{f(\vec{x}+h\vec{v})-f(\vec{x})}{h}\approx\frac{\nabla f(\vec{x})\cdot h\vec{v}}{h}=\nabla f(\vec{x})\cdot\vec{v}. $$ (Obviously, there's some formalization needed here; but, I hope this gives you the idea!)

Now, if you recall, a dot product $\vec{v}\cdot\vec{w}$ can be written as $$ \vec{v}\cdot\vec{w}=\|\vec{v}\|\,\|\vec{w}\|\cos\theta, $$ where $\theta$ is the angle between $\vec{v}$ and $\vec{w}$. In our specific case, this says $$ D_{\vec{v}}f(\vec{x})=\|\nabla f(\vec{x})\|\cos\theta, $$ where $\theta$ is the angle between $\vec{v}$ and $\nabla f$. The only part of this expression that depends on $\vec{v}$ is $\cos\theta$; so, we maximize the directional derivative by maximizing $\cos\theta$, and minimize the directional derivative by minimizing $\cos\theta$. These happen when $\theta=0$ and $\theta=\pi$, respectively - corresponding to $v$ being either in the same direction as $\nabla f$ or the opposite direction.

  • $\begingroup$ Thanks for answer but that kinda missed my questions. I thought gradient was defined as having the direction of maximum increase. So using the gradient to prove that gradient vector is in direction of maximum increase serves no purpose. I need to know why we can use partial derivatives as components in gradient vector and the reason behind it. $\endgroup$ – user1316208 Jun 21 '13 at 14:20
  • $\begingroup$ I always see the gradient defined as the vector of partial derivatives. Let me see if I can show you how that relates to directional derivatives. $\endgroup$ – Nick Peterson Jun 21 '13 at 14:21
  • $\begingroup$ I suppose it doesnt matter, its the equality of those two things that i dont understand. Why the partial derivatives give away the direction of maximum increase in this manner $\endgroup$ – user1316208 Jun 21 '13 at 14:28
  • $\begingroup$ Does that help at all? $\endgroup$ – Nick Peterson Jun 21 '13 at 14:33
  • $\begingroup$ No, I still cannot see the reason for such relation between partial derivatives and gradient vector. Perhaps i lack proper mathematical foundation to understand. Thanks for trying anyway $\endgroup$ – user1316208 Jun 21 '13 at 14:46

Let's assume that we have a function of two variables $f(x_1,x_2)$. The change in any direction can be expressed as $$df=f(x_1+h_1,x_2+h_2)-f(x_1,x_2)$$ By using Taylor expansion for $f(x_1+h_1,x_2+h_2)$ the change becomes $$df=f(x_1,x_2)+\frac{\partial f(x_1,x_2)}{\partial x_1}h_1+\frac{\partial f(x_1,x_2)}{\partial x_2}h_2-f(x_1,x_2)$$ The rate of change can be written as $$\frac{df}{\sqrt{h_1^2+h_2^2}}=\frac{\partial f(x_1,x_2)}{\partial x_1}\frac{h_1}{\sqrt{h_1^2+h_2^2}}+\frac{\partial f(x_1,x_2)}{\partial x_2}\frac{h_2}{\sqrt{h_1^2+h_2^2}}$$ Since we want to maximize the rate of change wrt $h_1$ and $h_2$ we set the first derivatives to zero $$\frac{\partial f(x_1,x_2)}{\partial x_1}\Bigg(\frac{1}{\sqrt{h_1^2+h_2^2}}-\frac{h_1^2}{(h_1^2+h_2^2)^{3/2}}\Bigg)-\frac{\partial f(x_1,x_2)}{\partial x_2}\frac{h_1h_2}{(h_1^2+h_2^2)^{3/2}}=0$$ $$-\frac{\partial f(x_1,x_2)}{\partial x_1}\frac{h_1h_2}{(h_1^2+h_2^2)^{3/2}}+\frac{\partial f(x_1,x_2)}{\partial x_2}\Bigg(\frac{1}{\sqrt{h_1^2+h_2^2}}-\frac{h_2^2}{(h_1^2+h_2^2)^{3/2}}\Bigg)=0$$ Multipliying both equations by $(h_1^2+h_2^2)^{3/2}$ $$\frac{\partial f(x_1,x_2)}{\partial x_1}h_2^2-\frac{\partial f(x_1,x_2)}{\partial x_2}h_1h_2=0$$ $$-\frac{\partial f(x_1,x_2)}{\partial x_1}h_1h_2+\frac{\partial f(x_1,x_2)}{\partial x_2}h_1^2=0$$ As can be seen the equations are satisfied if $$h_1=\frac{\partial f(x_1,x_2)}{\partial x_1}\text{ and } h_2=\frac{\partial f(x_1,x_2)}{\partial x_2}$$ which is simply the gradient.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.