# Hessian of second-order function of a matrix -- trace(A X^T B X C)

How do I find the Hessian (second-derivative) of $$f(X) = tr(A X^T B X C)$$?

Using differential notation, this has the following first differential:

$$d tr(A X^T B X C) = tr((CAX^TB + A^T C^T X^T B^T) dX)$$.

This straightforwardly leads to the first derivative $$(CAX^TB + A^T C^T X^T B^T)^T = B^T X A^TC^T + BXCA$$.

If I've done my algebra correctly, the second differential is

$$d^2 tr(A X^T B X C) = tr(CAdX^T B dX + A^TC^TdX^T B^T dX)$$.

How do I obtain the Hessian from this? I believe I need to somehow put this into the form $$vec(dX)^T H vec(dX)$$, from which I can conclude that the Hessian is $$H$$.

$$\def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\vc#1{\op{vec}\LR{#1}} \def\trace#1{\op{Tr}\LR{#1}} \def\frob#1{\left\| #1 \right\|_F} \def\qiq{\quad\implies\quad} \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}}$$Your gradient calculation is indeed correct. To find the Hessian...
First flatten $$X$$ and the gradient matrix into vectors \eqalign{ x &= \vc{X} \\ G &= B^TXA^TC^T + BXCA \\ g &= \vc{G} \\&= \LR{CA\otimes B^T}x + \LR{A^TC^T\otimes B}x \\ } Then calculate the gradient of the gradient (i.e. the Hessian) \eqalign{ dg &= \LR{CA\otimes B^T}dx + \LR{A^TC^T\otimes B}dx \\ \grad{g}{x} &= \LR{CA\otimes B^T} + \LR{A^TC^T\otimes B} \;\equiv\; Hessian \\ }