1
$\begingroup$

Let $A$ be a real-valued matrix and $$f(A) := \lVert A \rVert_F = \sqrt{A:A} = \sqrt{\mbox{tr}(A^TA)}$$ be its Frobenius norm. What is $\frac{\partial^2f}{\partial A^2}$?

For the squared Frobenius norm, $f^2$, I found that $\frac{\partial{f^2}}{\partial A}=2A$ and $\frac{\partial^2 f^2}{\partial A^2}=2 I\otimes I$. For the case of interest I was able to compute $\frac{\partial f}{\partial A}=A/\lVert A\rVert_F$, however I am not able to deal with the square root in the denominator for the second derivative and would appreciate any advice.

$\endgroup$
  • $\begingroup$ The gradient of scalar field $f^2$ is a function whose input and output are matrices. The 2nd derivative would be a $4$-dimensional matrix, for you need to differentiate every entry of the output matrix with respect to every entry of the input matrix. $\endgroup$ – Rodrigo de Azevedo May 25 '19 at 9:25
1
$\begingroup$

The matrix inner product $\,A:B={\rm Tr}(A^TB)$ can be used to write the Frobenius norm as $$\eqalign{ f^2 &= \|A\|_F^2 = A:A \cr }$$ Find the differential and gradient of this expression. $$\eqalign{ 2f\,df &= 2A:dA \cr df &= \frac{A}{f}:dA \cr G=\frac{\partial f}{\partial A} &= \frac{A}{f} \cr }$$ This matrix gradient can be vectorized. $$\eqalign{ g &= {\rm vec}(G) = \frac{{\rm vec}(A)}{f} = \frac{a}{f} \cr }$$ Note the alternate expressions for the differential of the norm. $$\eqalign{ df &= G:dA \cr df &= g^Tda \cr }$$ Now find the Hessian. $$\eqalign{ dg &= \frac{da}{f} - \frac{a}{f^2}\,df \cr &= \frac{da}{f} - \frac{a}{f^2}\,(g^Tda) \cr &= \frac{1}{f}\Big(I-gg^T\Big)\,da \cr H=\frac{\partial g}{\partial a} &= \frac{1}{f}\Big(I-gg^T\Big) \,\,= \frac{\partial^2f}{\partial a\,\partial a^T} \cr }$$ The Hessian can also be written entirely in terms of $a$ $$\eqalign{ H &= \frac{1}{f^3}\Big(f^2I-aa^T\Big) }$$ Without vectorization, the Hessian is a 4th order tensor, which can be written in component form as $$\eqalign{ H_{ijkl} &= \frac{\partial^2f}{\partial A_{ij}\,\partial A_{kl}} \cr &= \frac{1}{f^3}\Big(f^2\delta_{ik}\delta_{jl}-A_{ij}A_{kl}\Big) }$$ Note the permuted order of the indices on the $\delta$ symbols.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.