Let $f(A):= A^\top A$ where $A$ is an $m \times n$ matrix. We want to find the derivative of $f$ with respect to $A$. By derivative we mean to find the Jacobian of all partial derivatives of $f(A)$ with respect to $A$. Here is how I proceed.

The Derivative of $f$ is the linear map $D f(A): X \to A^\top X + X^\top A$. Let $K$ be the commutation matrix such that $K\operatorname{vec}(X^\top A) = \operatorname{vec}(A^\top X)$. Then,

\begin{align} \operatorname{vec}(A^\top X + X^\top A) & = \operatorname{vec}(A^\top X) + \operatorname{vec}(X^\top A) \\ & = (I_n\otimes A^\top) \operatorname{vec}(X) + \operatorname{vec}(X^\top A) \\ & = I_n (\otimes A^\top) \operatorname{vec}(X) + K_{n,n} \operatorname{vec}(A^\top X) \\ & = (I_n \otimes A^\top) \operatorname{vec}(X) + K_{n, n} (I_n \otimes A^\top) \operatorname{vec}(X) \end{align}

It now follows that \begin{align} \frac{\partial f}{\partial A} & = (I_n \otimes A^\top) + K_{n, n} (I_n \otimes A^\top) \end{align}

In here I am using the fact that $\operatorname{vec}(AXB) = (B^\top \otimes A)\operatorname{vec}(X)$ where $\operatorname{vec}$ is the vectorization operator.

I was inspired by this answer and the corresponding equation under the section Differentials of Quadratic Products on this webpage

My Questions:

  1. Is this approach correct?. If not how do I go about finding the desired derivative?

  2. Where can I find references regarding this type of manipulation?. (I don't mean this particular manipulation, but a reference for derivatives of matrices in general). I looked on Horn and Johnson Matrix Analysis, but a 'commutation matrix' is nowhere to be found. When I say reference, I mean a rigorous linear algebraic exposition.


Take the differential of the expression $$\eqalign{ F &= A^TA \cr dF &= dA^T\,A + A^T\,dA \cr }$$ At this point, you can either use vectorizations $$\eqalign{ {\rm vec}(dF) &= {\rm vec}(dA^T\,A) + {\rm vec}(A^T\,dA) \cr df &= (A^T\otimes I)(K\,da) + (I\otimes A^T)\,da \cr \frac{\partial f}{\partial a} &= (A^T\otimes I)K + (I\otimes A^T) \cr }$$ or tensor methods $$\eqalign{ dF &= (I{\mathcal E}A^T):({\mathcal K}:dA) + (A^T{\mathcal E}I):dA \cr \frac{\partial F}{\partial A} &= ({\mathcal E}A^T):{\mathcal K} + A^T{\mathcal E} \cr }$$ where a colon represents the double-contraction product, i.e. $$(X:{\mathcal E})_{kl} = \sum_{ij} X_{ij} {\mathcal E}_{ijkl} $$ while juxtapositions represent single-contractions $$(X{\mathcal E}Y)_{ikmr} = \sum_{jp} X_{ij} {\mathcal E}_{jkmp} Y_{pr} $$ The isotropic 4th order tensors have components $$\eqalign{ {\mathcal E}_{ijkl} &= \delta_{ik} \delta_{jl} \cr {\mathcal K}_{ijkl} &= \delta_{il} \delta_{jk} \cr\cr }$$ For references, try
"Matrix Differential Calculus" by Magnus and Neudecker
"Complex-Valued Matrix Derivatives" by Are Hjorungnes


You are close. By my calculation (checked on a $2\,x\,2$ example) $$\frac{\partial }{{\partial \underline {\overline {\bf{A}} } }}\left( {{{\underline {\overline {\bf{A}} } }^T}\underline {\overline {\bf{A}} } } \right) = \left( {{{\underline {\overline {\bf{I}} } }_{\left[ n \right]}} \otimes {{\underline {\overline {\bf{A}} } }^T}} \right) + \left( {{{\underline {\overline {\bf{A}} } }^T} \otimes {{\underline {\overline {\bf{I}} } }_{\left[ n \right]}}} \right){\underline {\overline {\bf{K}} } _{\left[ {m,n} \right]}}$$ Derivation: $$\frac{\partial }{{\partial \underline {\overline {\bf{A}} } }}\left( {{{\underline {\overline {\bf{A}} } }^T}\underline {\overline {\bf{A}} } } \right) = {\left. {\frac{\partial }{{\partial \underline {\overline {\bf{A}} } }}\left( {{{\underline {\overline {\bf{A}} } }^T}\underline {\overline {\bf{A}} } } \right)} \right|_{{{\underline {\overline {\bf{A}} } }^T}{\rm{ constant}}}} + {\left. {\frac{\partial }{{\partial \underline {\overline {\bf{A}} } }}\left( {{{\underline {\overline {\bf{A}} } }^T}\underline {\overline {\bf{A}} } } \right)} \right|_{\underline {\overline {\bf{A}} } {\rm{ constant}}}}$$ For the first term $${\underline {\overline {\bf{A}} } ^T}\underline {\overline {\bf{A}} } = {\underline {\overline {\bf{A}} } ^T}\underline {\overline {\bf{A}} } \,{\underline {\overline {\bf{I}} } _{\left[ n \right]}} = \left( {{{\underline {\overline {\bf{I}} } }_{\left[ n \right]}} \otimes {{\underline {\overline {\bf{A}} } }^T}} \right){\rm{vec}}\left( {\underline {\overline {\bf{A}} } } \right)$$ so that $${\left. {\frac{\partial }{{\partial \underline {\overline {\bf{A}} } }}\left( {{{\underline {\overline {\bf{A}} } }^T}\underline {\overline {\bf{A}} } } \right)} \right|_{{{\underline {\overline {\bf{A}} } }^T}{\rm{ constant}}}} = \left( {{{\underline {\overline {\bf{I}} } }_{\left[ n \right]}} \otimes {{\underline {\overline {\bf{A}} } }^T}} \right)$$ For the second term $${\underline {\overline {\bf{A}} } ^T}\underline {\overline {\bf{A}} } = {\underline {\overline {\bf{I}} } _{\left[ n \right]}}{\underline {\overline {\bf{A}} } ^T}\underline {\overline {\bf{A}} } = \left( {{{\underline {\overline {\bf{A}} } }^T} \otimes {{\underline {\overline {\bf{I}} } }_{\left[ n \right]}}} \right){\rm{vec}}\left( {{{\underline {\overline {\bf{A}} } }^T}} \right) = \left( {{{\underline {\overline {\bf{A}} } }^T} \otimes {{\underline {\overline {\bf{I}} } }_{\left[ n \right]}}} \right){\underline {\overline {\bf{K}} } _{\left[ {m,n} \right]}}{\rm{vec}}\left( {\underline {\overline {\bf{A}} } } \right)$$ so that $${\left. {\frac{\partial }{{\partial \underline {\overline {\bf{A}} } }}\left( {{{\underline {\overline {\bf{A}} } }^T}\underline {\overline {\bf{A}} } } \right)} \right|_{\underline {\overline {\bf{A}} } {\rm{ constant}}}} = \left( {{{\underline {\overline {\bf{A}} } }^T} \otimes {{\underline {\overline {\bf{I}} } }_{\left[ n \right]}}} \right){\underline {\overline {\bf{K}} } _{\left[ {m,n} \right]}}$$ I found it a challenge to stitch together all the different results required to do this type of calculation proficiently (which I needed to compute the Jacobian determinant of SVD transformations). One very useful reference that dealt with elimination and commutation matrices is:

Magnus, J., and Neudecker, H., “The Elimination Matrix: Some Lemmas and Applications,” SIAM J. on Algebraic. and Discrete Meth., V. 1, Issue 4, pp 422-449, Dec. 1980.

However, this doesn’t cover anything to do with the calculus side of things. I ended up compiling my own list of useful results, which (for the real case) can be found here in Section 3. The fact that it is Rev 8 gives you a sense of how easy it is to mess things up.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.