5
$\begingroup$

Given $f : \mbox{Sym} (n) \to \mathbb{R}$, let function $g: \mathbb{R}^{k ,n} \to \mathbb{R}$ be such that

$$g(X) := f \left(X^T X \right)$$

What is the Hessian of $g$ in terms of the Hessian of $f$?


Vectorizing $X$ and directly computing for the Hessian definitely works, but the notation is quite messy and I spent a lot of time on it but still can't get a clear solution. For now, I want to use the gradient which is $X(\nabla f(X^TX)^T + \nabla f(X^TX))$ to take its derivative and get the linear map representation of the hessian. But I'm having trouble taking the derivative of $\nabla f(X^TX)$. Suppose $Hf : Sym(n) \to Sym(n)$ is the linear map representation of the Hessian of $f$, then is $D(\nabla f(X^TX))$ just the map $V \to Hf(X^TX)V^TX+Hf(X^TX)X^TV$? If so, then the hessian of $g$ at $X$ would be the linear map on $\mathbb{R}^{k,n}$ that $$V \to X(D(\nabla f(X^TX))(V)^T + D(\nabla f(X^TX))(V))+V(\nabla f(X^TX)^T+\nabla f(X^TX))$$

But I'm not really sure of this. Any remarks will be much appreciated.

$\endgroup$
3

1 Answer 1

3
$\begingroup$

$ \def\a{\phi} \def\c#1{\color{red}{#1}} \def\wh{\widehat} \def\Ghat{\wh{G}} \def\E{{\cal E}} \def\F{{\cal F}} \def\H{{\cal H}} \def\K{{\cal K}} \def\J{{\cal J}} \def\Hhat{\wh{\cal H}} \def\L{\left} \def\R{\right} \def\n{\nabla} \def\o{{\tt1}} \def\p{\partial} \def\LR#1{\L(#1\R)} \def\trace#1{\operatorname{Tr}\LR{#1}} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\hess#1#2#3{\frac{\p^2 #1}{\p #2\,\p #3}} $For typing convenience, define the matrix variable $$A=X^TX \quad\implies\quad dA=dX^TX+X^TdX$$ Let's also define the Frobenius product, which is a convenient notation for the trace $$\eqalign{ A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{A^TB} \\ A:A &= \big\|A\big\|^2_F \\ }$$ We are interested in the scalar quantity $$\a = g(X) = f(A)$$ The gradient and Hessian of $f(A)\,$ are $2^{nd}$ and $4^{th}$ order tensors (assumed as given) $$\eqalign{ G_{ij} &= \grad{\a}{A_{ij}} \quad&\iff\quad G = \grad{\a}{A} \\ \H_{ijk\ell} &= \grad{G_{ij}}{A_{k\ell}} \quad&\iff\quad\H = \grad{G}{A} \\ }$$ Some additional $4^{th}$ order tensors will prove useful $$\eqalign{ \K_{ijk\ell} &= \H_{jik\ell} \quad&\iff\quad&\K = \grad{G^T}{A} \\ \E_{ijkl} &= \delta_{ik}\delta_{j\ell} \quad&\iff\quad&\E = \grad{A}{A} \\ \F_{ijkl} &= \delta_{i\ell}\delta_{jk} \quad&\iff\quad&\F = \grad{A^T}{A} \\ &&&\J = \Big(\H + \K\Big) \\ }$$ Now calculate the gradient $(\Ghat)$ with respect to $X$ $$\eqalign{ d\a &= G:dA \\ &= G:\LR{dX^TX+X^TdX} \\ &= X\LR{G+G^T}:dX \\ \grad{\a}{X} &= X\LR{G+G^T} \quad\doteq\quad \Ghat \\ }$$ and the Hessian $(\Hhat)$ wrt $X$ $$\eqalign{ d\Ghat &= dX\LR{G+G^T} + X\LR{dG+dG^T} \\ &= \Big(\E\cdot\LR{G+G^T}\!\Big):dX + X\cdot\LR{\J:dA} \\ &= \Big(\E\cdot\LR{G+G^T}\!\Big):dX + X\cdot\J:\LR{dX^TX+X^TdX} \\ &= \LR{\E\cdot G +\E\cdot G^T}\c{:dX} + \LR{X\cdot\J\cdot X^T}:\F \c{:dX} + \LR{X\cdot\J}:\LR{X^T\!\cdot\E}\c{:dX} \\ \grad{\Ghat}{X} &= \LR{\E\cdot G +\E\cdot G^T} + \LR{X\cdot\J\cdot X^T}:\F + \LR{X\cdot\J}:\LR{X^T\!\cdot\E} \quad\doteq\quad \Hhat \\ }$$ These results provide two expressions for the second-order Taylor series $$\eqalign{ \a &= f(A_0) + G :(A-A_0) + \tfrac 12(A-A_0):\H :(A-A_0) \\ &= g(X_0) + \Ghat:(X-X_0) + \tfrac 12(X-X_0):\Hhat:(X-X_0) \\ \\ }$$


The properties of the underlying trace function allow the terms in a Frobenius between matrices to be rearranged in many equivalent ways, e.g. $$\eqalign{ A:B &= B:A \\ A:B &= A^T:B^T \\ A:BX &= B^TA:X = AX^T:B \\ }$$ When applied to tensors the Frobenius product corresponds to a double-dot product, just as the matrix product corresponds to a single-dot product. For example, $$\eqalign{ \J:A &\implies \sum_{k=1}^n\sum_{\ell=1}^n \J_{ijk\ell}A_{k\ell} \\ A\cdot\J\cdot B &\implies \sum_{i=1}^n\sum_{\ell=1}^n A_{pi}\J_{ijk\ell}B_{\ell s} \\\\ }$$


Update

The error pointed out by Koncopd necessitated the introduction of yet another tensor $(\F)$.
So this is probably a good time to demonstrate the algebraic properties of these tensors. $$\eqalign{ \E:A &= A:\E &= A \qquad&\big({\rm identity\,operator}\big) \\ \F:A &= A:\F &= A^T &\big({\rm transpose\,operator}\big) \\ }$$ $$\eqalign{ &A\cdot X\cdot B &= \LR{A\cdot\E\cdot B^T}:X \qquad\qquad\qquad\qquad \\ &A\cdot X^T\!\cdot B &= \LR{A\cdot\E\cdot B^T}:\F:X \\ }$$

$\endgroup$
2
  • 1
    $\begingroup$ Hm, is this really correct $dG={\cal H}:dX$? Shouldn't it be $dG={\cal H}:dA$? $\endgroup$
    – Koncopd
    Aug 10, 2021 at 9:56
  • $\begingroup$ @Koncopd Thank you for spotting that error. Although the result isn't as pretty as it used to be, at least it's correct. $\endgroup$
    – greg
    Aug 11, 2021 at 16:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .