What is the Hessian of $X \mapsto f \left( X^T X \right)$ in terms of the Hessian of $f$? Given $f : \mbox{Sym} (n) \to \mathbb{R}$, let function $g: \mathbb{R}^{k ,n} \to \mathbb{R}$ be such that
$$g(X) := f \left(X^T X \right)$$
What is the Hessian of $g$ in terms of the Hessian of $f$?

Vectorizing $X$ and directly computing for the Hessian definitely works, but the notation is quite messy and I spent a lot of time on it but still can't get a clear solution. For now, I want to use the gradient which is $X(\nabla f(X^TX)^T + \nabla f(X^TX))$ to take its derivative and get the linear map representation of the hessian. But I'm having trouble taking the derivative of $\nabla f(X^TX)$. Suppose $Hf : Sym(n) \to Sym(n)$ is the linear map representation of the Hessian of $f$, then is $D(\nabla f(X^TX))$ just the map $V \to Hf(X^TX)V^TX+Hf(X^TX)X^TV$? If so, then the hessian of $g$ at $X$ would be the linear map on $\mathbb{R}^{k,n}$ that $$V \to X(D(\nabla f(X^TX))(V)^T + D(\nabla f(X^TX))(V))+V(\nabla f(X^TX)^T+\nabla f(X^TX))$$
But I'm not really sure of this. Any remarks will be much appreciated.
 A: $
\def\a{\phi}
\def\c#1{\color{red}{#1}}
\def\wh{\widehat}
\def\Ghat{\wh{G}}
\def\E{{\cal E}}
\def\F{{\cal F}}
\def\H{{\cal H}}
\def\K{{\cal K}}
\def\J{{\cal J}}
\def\Hhat{\wh{\cal H}}
\def\L{\left}
\def\R{\right}
\def\n{\nabla}
\def\o{{\tt1}}
\def\p{\partial}
\def\LR#1{\L(#1\R)}
\def\trace#1{\operatorname{Tr}\LR{#1}}
\def\grad#1#2{\frac{\p #1}{\p #2}}
\def\hess#1#2#3{\frac{\p^2 #1}{\p #2\,\p #3}}
$For typing convenience, define the matrix variable
$$A=X^TX \quad\implies\quad dA=dX^TX+X^TdX$$
Let's also define the Frobenius product, which is a convenient notation for the trace
$$\eqalign{
A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{A^TB} \\
A:A &= \big\|A\big\|^2_F \\
}$$
We are interested in the scalar quantity
$$\a = g(X) = f(A)$$
The gradient and Hessian of $f(A)\,$ are $2^{nd}$ and $4^{th}$
order tensors (assumed as given)
$$\eqalign{
G_{ij} &= \grad{\a}{A_{ij}} 
  \quad&\iff\quad G = \grad{\a}{A} \\
\H_{ijk\ell} &= \grad{G_{ij}}{A_{k\ell}} 
  \quad&\iff\quad\H = \grad{G}{A} \\
}$$
Some additional $4^{th}$ order tensors will prove useful
$$\eqalign{
\K_{ijk\ell} &= \H_{jik\ell} 
  \quad&\iff\quad&\K = \grad{G^T}{A} \\
\E_{ijkl} &= \delta_{ik}\delta_{j\ell} 
  \quad&\iff\quad&\E = \grad{A}{A} \\
\F_{ijkl} &= \delta_{i\ell}\delta_{jk} 
  \quad&\iff\quad&\F = \grad{A^T}{A} \\
&&&\J = \Big(\H + \K\Big) \\
}$$
Now calculate the gradient $(\Ghat)$ with respect to $X$
$$\eqalign{
d\a &= G:dA \\
 &= G:\LR{dX^TX+X^TdX} \\
 &= X\LR{G+G^T}:dX \\
\grad{\a}{X} &= X\LR{G+G^T} \quad\doteq\quad \Ghat \\
}$$
and the Hessian $(\Hhat)$ wrt $X$
$$\eqalign{
d\Ghat &= dX\LR{G+G^T} + X\LR{dG+dG^T} \\
  &= \Big(\E\cdot\LR{G+G^T}\!\Big):dX + X\cdot\LR{\J:dA} \\
  &= \Big(\E\cdot\LR{G+G^T}\!\Big):dX + X\cdot\J:\LR{dX^TX+X^TdX} \\
  &= \LR{\E\cdot G +\E\cdot G^T}\c{:dX}
   + \LR{X\cdot\J\cdot X^T}:\F \c{:dX}
   + \LR{X\cdot\J}:\LR{X^T\!\cdot\E}\c{:dX} \\
\grad{\Ghat}{X}
 &= \LR{\E\cdot G +\E\cdot G^T}
  + \LR{X\cdot\J\cdot X^T}:\F
  + \LR{X\cdot\J}:\LR{X^T\!\cdot\E} 
  \quad\doteq\quad \Hhat \\
}$$
These results provide two expressions for the second-order Taylor series
$$\eqalign{
\a
  &= f(A_0) +  G   :(A-A_0) + \tfrac 12(A-A_0):\H   :(A-A_0) \\
  &= g(X_0) + \Ghat:(X-X_0) + \tfrac 12(X-X_0):\Hhat:(X-X_0) \\
\\
}$$

The properties of the underlying trace function allow the terms in a Frobenius between matrices to be rearranged in many equivalent ways, e.g.
$$\eqalign{
A:B &= B:A \\
A:B &= A^T:B^T \\
A:BX &= B^TA:X = AX^T:B \\
}$$
When applied to tensors the Frobenius product corresponds to a double-dot product, just as the matrix product corresponds to a single-dot product. For example,
$$\eqalign{
\J:A &\implies \sum_{k=1}^n\sum_{\ell=1}^n \J_{ijk\ell}A_{k\ell} \\
A\cdot\J\cdot B &\implies \sum_{i=1}^n\sum_{\ell=1}^n A_{pi}\J_{ijk\ell}B_{\ell s} \\\\
}$$

Update
The error pointed out by Koncopd necessitated the introduction of yet another tensor $(\F)$.
So this is probably a good time to demonstrate the algebraic properties of these tensors.
$$\eqalign{
\E:A &= A:\E &= A  \qquad&\big({\rm identity\,operator}\big) \\
\F:A &= A:\F &= A^T &\big({\rm transpose\,operator}\big) \\
}$$
$$\eqalign{
&A\cdot X\cdot B &= \LR{A\cdot\E\cdot B^T}:X \qquad\qquad\qquad\qquad \\
&A\cdot X^T\!\cdot B &= \LR{A\cdot\E\cdot B^T}:\F:X \\
}$$
