Matrix Calculus - Composition with Differentiable Function I'm fitting a model to some data, and am trying to take the following derivative:
$$\frac{\partial}{\partial V}\|U \phi(VX)-Y\|_F^2$$
where $\phi$ is a differentiable function applied entry-wise.
From the matrix cookbook, I've (doubtfully) gotten to $\operatorname{Tr}((2U^T(U \phi(VX)-Y))^T\frac{\partial}{\partial V}\phi(VX))$, but am unsure how to proceed further.
 A: Define the matrices
$$\eqalign{
Z &= VX,\quad F=\phi(Z),\quad G=\phi'(Z) \\
}$$
where $\phi'$ is the derivative of $\phi$ and is also applied element-wise.
Use these to calculate the gradient of the cost function.
$$\eqalign{
{\cal L} &= \|UF-Y\|^2 \\&= (UF-Y):(UF-Y) \\
d{\cal L} &= 2(UF-Y):U\,dF \\
 &= 2U^T(UF-Y):dF \\
 &= 2U^T(UF-Y):G\odot dZ \\
 &= 2G\odot(U^TUF-U^TY):dZ \\
 &= 2G\odot(U^TUF-U^TY):dV\,X \\
 &= 2\Big(G\odot(U^TUF-U^TY)\Big)X^T:dV \\
\frac{\partial{\cal L}}{\partial V} &= 2\Big(G\odot(U^TUF-U^TY)\Big)X^T \\
}$$
where the symbol $(\odot)$ denotes the elementwise/Hadamard product, and the symbol $(:)$ represents the trace/Frobenius product, i.e.
$$A:B = {\rm Tr}(A^TB)$$
The Hadamard and Frobeius products commute with themselves and each other.
$$\eqalign{
A\odot B &= B\odot A \\
A : B &= B : A \\
(A\odot B):C &= A:(B\odot C) \\
}$$
Further the cyclic property of the trace allows terms in its product to be rearranged, e.g.
$$\eqalign{
A:BC &= AC^T:B \;=\; CA^T:B^T \;=\; etc \\
}$$
