# gradient approximation to the change of function

Say $$E = f(X) \\ \text{when}\ X \to X+\delta \\\text{where}\ \|\delta\| \to 0\ \text{is a vector}$$ then $$\Delta E \approx f(X) +\delta^T \nabla_X f(X)$$

Is this correct ? Then my question is , by the definition of gradient, the gradient should be the direction which increases your function value the most. But now I am not moving my X along with that direction. Howcome the equation above is correct ?

Recall from a 1-D derivative, the gradient tells you what increase you will get if you "move in this direction that I tell you"

But now I am not moving along with the gradient direction. Why is this still the approximation for $$\Delta E$$

My thought is that the equation should be $$\Delta E = f(X) +\delta^T \nabla_{\delta}f(X)$$

Am I missing something ?

-
Is $f: \Bbb R^n \to \Bbb R$ (or $\Bbb C^n \to \Bbb C$)? In that case, probably $\Delta E$ should be $\delta^T \nabla_X f(X)$. –  Lord_Farin Oct 15 '12 at 11:44

Recall the gradient is defined (for $f: \Bbb R^n \to \Bbb R$) as:

$$\nabla_X f = \nabla f(X) := \begin{pmatrix}\partial_1 f(X) \\ \vdots \\ \partial_n f(X)\end{pmatrix}$$

where $\partial_i f := \dfrac{\partial f}{\partial x_i}$ WRT the standard basis on $\Bbb R^n$. It is the transpose of the Jacobian matrix for real-valued functions $f$.

With this in mind, $\delta^T \nabla f(X) = \left<\delta, \nabla f(X)\right>$ is what we obtain by "linearly approximating $f$ in the direction of $\delta$ by the length of $\delta$".

This appears to be a correct first-order approximation of the change if $\|\delta\| \to 0$.

Your confusion seems to arise from the expression $\nabla_X f(X)$, which is not correct.

-
I think we do not need to the term f(X) in $\Delta E$. Am I right? –  muratguner Oct 15 '12 at 21:53
I would say so; as you can see in the question's comments, and also in my answer, I have suggested deleting that term and not included it, respectively. OP is irresponsive, though. –  Lord_Farin Oct 15 '12 at 22:31
thank you. yes f(X) :R^n -> R since I am dealing with optimization problem. However, my thought is we should compute the "derivative" along the direction I am gonna move, instead of the gradient (which is the direction which allows us to increase our f(X) the most). Because I think my approximation to $$\Delta E$$ along with some direction $$\delta$$ should be $$\delta \times rate\ of\ change\ along\ with\ \delta$$ which is not the same as gradient –  Jing Oct 16 '12 at 0:10
Please do note that $\delta^T \nabla f(X) = Df(X)(\delta)$ in the notation of multivariable calculus; alternatively, it's $\|\delta\| \cdot D_{\hat \delta} f(X)$ where $\hat \delta = \dfrac{\delta}{\|\delta\|}$ is the direction of $\delta$. The gradient does not only determine the direction in which $f(X)$ increases the most, it also allows to compute the rate of change in any other direction (that these two coincide is a virtue of the fact that the inner product of two vectors is largest if they are collinear). Please read this WikiPedia thoroughly. –  Lord_Farin Oct 16 '12 at 6:07