0
$\begingroup$

Consider a (continuously differentiable as many times as you need it) function $f(x)$: $\mathbb R^n \Rightarrow \mathbb R$.

Let $g(x)$ = gradient of $f(x)$ w.r.t. $x$.

Let $H(x)$ = Hessian of $f(x)$ w.r.t. $x$.

Then gradient w.r.t. $x$ of $g(x)^T g(x) = 2 H(x) g(x)$.

I believe Hessian w.r.t. $x$ of $g(x)^T g(x) = 2 H(x)^2 +$ higher order term(s).

I have a guess that higher order term(s) is a matrix of zeros when $g(x)$ is the vector of zeros, but not sure about that. Are those higher order term(s) the result of some kind of higher order tensor operation on $H(x)$ combined with $g(x)$, such that they come out to the matrix of zeros when $g(x) = $ vector of zeros?

Is there a way to calculate the higher order term(s) purely in terms of $H(x)$ and $g(x)$, or do I need some kind of 3rd (tensor or whatever) derivative of $f(x)$ not expressible (computable) in terms of $H$ and $g$?

I am trying to determine the correct (or less preferably "good enough", at least when gradient is near the vector of zeros) expression to use in an optimization algorithm for the Hessian of $g(x)^T g(x)$, having available the values of $H(x)$ and $g(x)$.

$\endgroup$

1 Answer 1

1
$\begingroup$

Let $\phi$ be the function $x\rightarrow < g,g>$ .

$D\phi(h)=2<Dg,g>(h) =2D^2f(g,h)=2h^THg$

and $D\phi= 2Hg$, where $H$, the hessian of $f$, is a symmetric matrix.

$D^2\phi(h,k)=2h^TD(Hg)(k)=2h^T(DH(g,k)+HDg(k))=$

$2D^3f(g,h,k)+2h^TH^2k$ (*).

Finally $D^2\phi=2H^2+2D^3f(g,.,.)$.

$D^3f$ is a symmetric 3-linear form; then $D^3f(g,.,.)$ is a symmetric matrix as $H^2$. In particular, the two terms in (*) are both of degree $2$ in $(h,k)$ and the first term is not negligible compared to the second one, except when $g(x)=0$. Of course, if $g(x)=0$, then $D^2 \phi=2H^2$.

EDIT. $D^3f$ is the symmetric tensor of the partial derivatives of $f$ of length $3$, that is $D^3f(g,h,k)=\sum_{r,s,t}\dfrac{\partial ^3f}{\partial x_r\partial x_s\partial x_t}g_rh_sk_t$. Thus $D^3f(g,.,.)$ is the symmetric matrix with $(s,t)$ entry $\sum_{r}\dfrac{\partial ^3f}{\partial x_r\partial x_s\partial x_t}g_r$.

Numerically you can approximate the third derivatives by discretizing $H$.

$\endgroup$
1
  • $\begingroup$ Thanks. Is $D^3f(g,.,.)$ in effect a 3rd derivative of some kind of $f(x)$? Is there any way to numerically calculate it from the numerical values of $H(x)$ and $g(x)$ (I'm thinking not, but trying to keep hope alive)? $\endgroup$ May 22, 2016 at 15:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .