# Derivative of scalar function with respect to matrix with vectors involved

I want to find derivative of a scalar function with respect to matrix $A$:

$E=\|f(Ax)\|^2$

Where $f(Ax)$ is a vector, say color of pixel at position $Ax$. How can I do that, given that I can compute derivative of $f$ with respect to its argument $\partial{f}/\partial{x}$?

I know that I can rewrite function $E$ like this

$E=f(Ax)^Tf(Ax)$

or

$E=tr(f(Ax)f(Ax)^T)$

Is there a way to find derivative using matrix-vector operations? E.g. without computing derivatives with respect to individual matrix elements. Is there a general analog of chain rule? Say for $g(f(Ax))$, where g is scalar and f is vector.

• Yes. It's called the Matrix Cookbook. Start reading around page 8. math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf – Christopher Carl Heckman Apr 17 '16 at 2:24
• @CarlHeckman I've already studied cookbook and did a lot of matrix calculus in the past, yet I don't see pretty solution. – DikobrAz Apr 17 '16 at 2:36
• I wasn't sure by your post. And yes, it does become nasty, because you get $\delta A$ and $(\delta A)^\top$ in your answer ... – Christopher Carl Heckman Apr 17 '16 at 2:45

For convenience, define the variables \eqalign{ y &= Ax,\,\,\,f = f(y),\,\,\, J = \frac{\partial f}{\partial y} \cr } Then write the function in terms of the Frobenius (:) Inner Product and take its differential \eqalign{ E &= f:f \cr\cr dE &= 2f:df \cr &= 2f:J\,dy \cr &= 2f:J\,dA\,x \cr &= 2J^Tfx^T:dA \cr } Since $dE = \big(\frac{\partial E}{\partial A}:dA\big),\,$ the gradient must be \eqalign{ \frac{\partial E}{\partial A} &= 2\,J^Tfx^T \cr }