# What is the Hessian of $X \mapsto f \left( X^T X \right)$ in terms of the Hessian of $f$?

Given $$f : \mbox{Sym} (n) \to \mathbb{R}$$, let function $$g: \mathbb{R}^{k ,n} \to \mathbb{R}$$ be such that

$$g(X) := f \left(X^T X \right)$$

What is the Hessian of $$g$$ in terms of the Hessian of $$f$$?

Vectorizing $$X$$ and directly computing for the Hessian definitely works, but the notation is quite messy and I spent a lot of time on it but still can't get a clear solution. For now, I want to use the gradient which is $$X(\nabla f(X^TX)^T + \nabla f(X^TX))$$ to take its derivative and get the linear map representation of the hessian. But I'm having trouble taking the derivative of $$\nabla f(X^TX)$$. Suppose $$Hf : Sym(n) \to Sym(n)$$ is the linear map representation of the Hessian of $$f$$, then is $$D(\nabla f(X^TX))$$ just the map $$V \to Hf(X^TX)V^TX+Hf(X^TX)X^TV$$? If so, then the hessian of $$g$$ at $$X$$ would be the linear map on $$\mathbb{R}^{k,n}$$ that $$V \to X(D(\nabla f(X^TX))(V)^T + D(\nabla f(X^TX))(V))+V(\nabla f(X^TX)^T+\nabla f(X^TX))$$

But I'm not really sure of this. Any remarks will be much appreciated.

• Jul 20, 2021 at 14:01
• The third paper (by Panda) is very good.
– lynn
Jul 23, 2021 at 5:55
• Shouldn't $f$ be twice differentiable? Aug 11, 2021 at 20:03

$$\def\a{\phi} \def\c#1{\color{red}{#1}} \def\wh{\widehat} \def\Ghat{\wh{G}} \def\E{{\cal E}} \def\F{{\cal F}} \def\H{{\cal H}} \def\K{{\cal K}} \def\J{{\cal J}} \def\Hhat{\wh{\cal H}} \def\L{\left} \def\R{\right} \def\n{\nabla} \def\o{{\tt1}} \def\p{\partial} \def\LR#1{\L(#1\R)} \def\trace#1{\operatorname{Tr}\LR{#1}} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\hess#1#2#3{\frac{\p^2 #1}{\p #2\,\p #3}}$$For typing convenience, define the matrix variable $$A=X^TX \quad\implies\quad dA=dX^TX+X^TdX$$ Let's also define the Frobenius product, which is a convenient notation for the trace \eqalign{ A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{A^TB} \\ A:A &= \big\|A\big\|^2_F \\ } We are interested in the scalar quantity $$\a = g(X) = f(A)$$ The gradient and Hessian of $$f(A)\,$$ are $$2^{nd}$$ and $$4^{th}$$ order tensors (assumed as given) \eqalign{ G_{ij} &= \grad{\a}{A_{ij}} \quad&\iff\quad G = \grad{\a}{A} \\ \H_{ijk\ell} &= \grad{G_{ij}}{A_{k\ell}} \quad&\iff\quad\H = \grad{G}{A} \\ } Some additional $$4^{th}$$ order tensors will prove useful \eqalign{ \K_{ijk\ell} &= \H_{jik\ell} \quad&\iff\quad&\K = \grad{G^T}{A} \\ \E_{ijkl} &= \delta_{ik}\delta_{j\ell} \quad&\iff\quad&\E = \grad{A}{A} \\ \F_{ijkl} &= \delta_{i\ell}\delta_{jk} \quad&\iff\quad&\F = \grad{A^T}{A} \\ &&&\J = \Big(\H + \K\Big) \\ } Now calculate the gradient $$(\Ghat)$$ with respect to $$X$$ \eqalign{ d\a &= G:dA \\ &= G:\LR{dX^TX+X^TdX} \\ &= X\LR{G+G^T}:dX \\ \grad{\a}{X} &= X\LR{G+G^T} \quad\doteq\quad \Ghat \\ } and the Hessian $$(\Hhat)$$ wrt $$X$$ \eqalign{ d\Ghat &= dX\LR{G+G^T} + X\LR{dG+dG^T} \\ &= \Big(\E\cdot\LR{G+G^T}\!\Big):dX + X\cdot\LR{\J:dA} \\ &= \Big(\E\cdot\LR{G+G^T}\!\Big):dX + X\cdot\J:\LR{dX^TX+X^TdX} \\ &= \LR{\E\cdot G +\E\cdot G^T}\c{:dX} + \LR{X\cdot\J\cdot X^T}:\F \c{:dX} + \LR{X\cdot\J}:\LR{X^T\!\cdot\E}\c{:dX} \\ \grad{\Ghat}{X} &= \LR{\E\cdot G +\E\cdot G^T} + \LR{X\cdot\J\cdot X^T}:\F + \LR{X\cdot\J}:\LR{X^T\!\cdot\E} \quad\doteq\quad \Hhat \\ } These results provide two expressions for the second-order Taylor series \eqalign{ \a &= f(A_0) + G :(A-A_0) + \tfrac 12(A-A_0):\H :(A-A_0) \\ &= g(X_0) + \Ghat:(X-X_0) + \tfrac 12(X-X_0):\Hhat:(X-X_0) \\ \\ }

The properties of the underlying trace function allow the terms in a Frobenius between matrices to be rearranged in many equivalent ways, e.g. \eqalign{ A:B &= B:A \\ A:B &= A^T:B^T \\ A:BX &= B^TA:X = AX^T:B \\ } When applied to tensors the Frobenius product corresponds to a double-dot product, just as the matrix product corresponds to a single-dot product. For example, \eqalign{ \J:A &\implies \sum_{k=1}^n\sum_{\ell=1}^n \J_{ijk\ell}A_{k\ell} \\ A\cdot\J\cdot B &\implies \sum_{i=1}^n\sum_{\ell=1}^n A_{pi}\J_{ijk\ell}B_{\ell s} \\\\ }

Update

The error pointed out by Koncopd necessitated the introduction of yet another tensor $$(\F)$$.
So this is probably a good time to demonstrate the algebraic properties of these tensors. \eqalign{ \E:A &= A:\E &= A \qquad&\big({\rm identity\,operator}\big) \\ \F:A &= A:\F &= A^T &\big({\rm transpose\,operator}\big) \\ } \eqalign{ &A\cdot X\cdot B &= \LR{A\cdot\E\cdot B^T}:X \qquad\qquad\qquad\qquad \\ &A\cdot X^T\!\cdot B &= \LR{A\cdot\E\cdot B^T}:\F:X \\ }

• Hm, is this really correct $dG={\cal H}:dX$? Shouldn't it be $dG={\cal H}:dA$? Aug 10, 2021 at 9:56
• @Koncopd Thank you for spotting that error. Although the result isn't as pretty as it used to be, at least it's correct.
– greg
Aug 11, 2021 at 16:52