# Taking matrix derivative $\| \left| \mathbf{X}\mathbf{W}\right|-\mathbf{1}_{n \times K} \| ^2_F$ with respect to W

I am trying to take the matrix derivative of the following function with respect to $$\bf W$$:

$$$$\| \left| \mathbf{X}\mathbf{W}\right|-\mathbf{1}_{n \times K} \| ^2_F \\$$$$

Where $$\mathbf{X}$$ is $$n \times d$$, $$\mathbf{W}$$ is $$d \times K$$ and $$\mathbf{1}_{n \times K}$$ is a marix with all elements one. $$\| \cdot \|_F$$ is the Frobenius norm and $$\left| \mathbf{X}\mathbf{W}\right|$$ is the element wise absolute value of $$\mathbf{X}\mathbf{W}$$.

Any helps is highly appreciated.

For typing convenience, define the matrices \eqalign{ Y &= XW \\ J &= 1_{n\times K} \qquad&({\rm all\,ones\,matrix}) \\ S &= {\rm sign}(Y) \\ A &= S\odot Y \qquad&({\rm absolute\,value\,of\,}Y) \\ B &= A-J \\ Y &= S\odot A \qquad&({\rm sign\,property}) \\ } where $$\odot$$ denotes the elementwise/Hadamard product and the sign function is applied element-wise. Use these new variables to rewrite the function, then calculate its gradient. \eqalign{ \phi &= \|B\|_F^2 \\&= B:B \\ d\phi &= 2B:dB \\ &= 2(A-J):dA \\ &= 2(A-J):S\odot dY \\ &= 2S\odot(A-J):dY \\ &= 2(Y-S):dY \\ &= 2(Y-S):X\,dW \\ &= 2X^T(Y-S):dW \\ \frac{\partial\phi}{\partial W} &= 2X^T(Y-S) \\ } where a colon denotes the trace/Frobenius product, i.e. \eqalign{ A:B = {\rm Tr}(A^TB) = {\rm Tr}(AB^T) = B:A } The cyclic property of the trace allows such products to be rearranged in various ways \eqalign{ A:BC &= B^TA:C \\ &= AC^T:B \\ } Finally, when $$(A,B,C)$$ are all the same size, their Hadamard and Frobenius products commute with each other \eqalign{ A:B\odot C &= A\odot B:C \\\\ } NB: When an element of $$\,Y$$ equals zero, the gradient is undefined. This behavior is similar to the derivative of $$\,|x|\,$$ in the scalar case.
• The answer from greg is great. However, in the last part of the explanation, the correct notation for a colon is as follows: \eqalign{ A:B = {\rm Tr}(A^TB) = {\rm Tr}(B^TA) = B:A } – Hadi Zavareh Aug 19 at 22:05