# Hessian-vector products

Can someone explain why this is true?

$$g(x + \Delta x) = g(x) + H(x) \Delta x$$

where g is the gradient of function f(x) with respect to x, and H is the hessian of f(x) with respect to x.

I would really appreciate a detailed derivation because I don't understand what it means to take the gradient/derivative of $$\Delta x$$ which represents the change in x.

Source towards the top here: https://justindomke.wordpress.com/2009/01/17/hessian-vector-products/

• Is it like the third term in the taylor expansion? en.m.wikipedia.org/wiki/…
– Emil
Nov 28, 2018 at 7:14
• Is the lhs supposed to be the gradient taken at a point or is it a scalar? The hessian is a second order tensor, so it should eat two dx to become a scalar, not one, I believe.
– Emil
Nov 28, 2018 at 7:22
• $\Delta x$ is a vector I believe Nov 28, 2018 at 7:58
• I don't know why you're talking about scalars, the Hessian H(x) is a vector and so is $\Delta x$. Is it because the source link is $g(x) = \frac{d}{dx} f(x)$ is using scalar notation? He intends this to be the gradient of a vector Nov 28, 2018 at 8:01
• user3180: to know if there is an error and if so if it is on lhs or rhs (hessian is a second order tensor not a vector, according to some definitions, or maybe even third order tensor)
– Emil
Nov 28, 2018 at 13:26

For a scalar function f,

$$f(x+dx) \approx f(x)+ \frac{d f(x)}{dx}* dx$$

Vector case [$$x$$ is (n,1) dimensional vector]:

$$g(x + \Delta x) \approx g(x) + g(\Delta x)$$

$$g(x) = \nabla f(x) = \begin{bmatrix} g_1{(x)} \\ g_2{(x)} \\ \vdots \\ g_n{(x)} \end{bmatrix}$$ $$\mid g_i{(x)} = \frac{\partial f(x)}{dx_i}$$

$$H(x) = \nabla g(x) = \begin{bmatrix} \nabla g_1{(x)}^T \\ \nabla g_2{(x)}^T \\ \vdots \\ \nabla g_n{(x)}^T \end{bmatrix}$$

$$g(x) +g(\Delta x) = g(x) + \Delta g(x)\approx g(x)+ \begin{bmatrix} \nabla g_1{(x)}^T \Delta x \\ \nabla g_2{(x)}^T \Delta x\\ \vdots \\ \nabla g_n{(x)}^T \Delta x \end{bmatrix}$$

Translation: $$g(\Delta x)$$ represents g with a small change in x as input, which is equivalent to $$\Delta g(x)$$ [the infinitesimal change in g(x)].

If you look at the last column vector, you will see that each row $$\nabla g_i(x)^T \Delta x$$ represents the (rate of change of $$g_i(x)$$/rate of change of x) dotted with (infinitesimal change in x) := infinitesimal change in $$g_i(x)$$. In aggregate, the column vector represents the infinitesimal change in g(x) := $$\Delta g(x)$$.