Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Suppose I have a function F($\vec{x})$ $x\subset R^{n}$, and a vector $\vec{g}$. I want to perturb $\vec{x}$ along the $\vec{g}$ vector, and see how the gradient changes as I move further along the $\vec{g}$ vector. So I'm trying to solve $\lim_{h\to 0}(\nabla F(\vec{x}+h*\vec{g})-\nabla F(\vec{x}))/h$. Note, I don't want to know the change in the function as h approaches 0, but the change in each gradient (so there should be as many terms in the answer to the above formula as there are dimensions in $\vec{x}$. I'm pretty sure there should be an efficient way to do this (linear in the number of parameters), since at worst case I can just use a small value for h and use finite differences. Likewise, I think we should be able to get higher derivatives along the vector also linear in the number of parameters, since we can also accomplish that linear in the number of parameters with finite differences.

My first thought was to set $\vec{x}(h)=\vec{x}_{o}+h*\vec{g}$, and then try to use the chain rule, so I end up with $F'(\vec{x}(h))*\vec{x}'(h)*dh$, but that quantity seems like it would be one dimensional ($\frac{dF}{dh}$), and not the change in the each gradient itself. Plus, I'm not sure that would even work for higher derivatives, since $F''(\vec{x})$ has $n^{2}$ terms, so we're no longer linear in the number of parameters.

Now I'm trying to see if I can derive some transformation of F that will change it to a new function that has as its derivatives the quantities I'm trying to calculate, but I'm not having much luck doing it that way either. In general, I'd like to be able to calculate the change in the gradients along h even if $\vec{x}(h)$ is a non-linear function of h (so we're following a curve instead of a line), but for the time being I'm keeping it simple and saying that $\vec{x}(h)$ is linear.

Is there some general way to find these quantities.

share|improve this question

1 Answer 1

If I understand correctly, you are trying to take the directional derivative of the gradient of $F$ along $g$. $(\nabla F)^T$ is just another (vector-valued) function of $x$, and its directional derivative along $g$ is the dot product of the gradient of $(\nabla F)^T$ with g, ie, $$HF g,$$ where $$HF = \nabla (\nabla F)^T = \left[\begin{array}{ccc}\frac{\partial^2 F}{\partial x_1^2} & \frac{\partial^2 F}{\partial x_1 \partial x_2} & \ldots\\ \frac{\partial^2 F}{\partial x_1 \partial x_2} & \frac{\partial^2F}{\partial x_2^2} & \ldots\\\vdots & \vdots & \ddots \end{array}\right]$$ is the Hessian of $F$.

Unfortunately, computing this matrix-vector product for general $F$ is roughly cubic in the dimension $n$, and not linear. In many applications, however, $HF$ is sparse, so that computing and using it remains preferable to taking finite differences of $\nabla f$ due to the former's greater accuracy.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.