# Gradient of $W \mapsto \left\| W^T W - I \right\|_F^2$

I have a function of the following form

$$J (W) = \left\| W^T W - I \right\|_F^2$$

where $$W$$ is a matrix and $$\left\| \cdot \right\|_F$$ is the Frobenius norm. How can I find the gradient $$\nabla_W J$$?

You can use the product rule for matrix differentiation, $$\frac{\partial\mathrm{trace}(f(X)G(X))}{\partial X} = \left. \frac{\partial\mathrm{trace}(f(X)G(Y))}{\partial X}+\frac{\partial\mathrm{trace}(f(Y)G(X))}{\partial X}\right|_{Y=X}$$ along with the simpler identity $$\frac{\partial\mathrm{trace}(AX)}{\partial X}=A^\top.$$

For your problem this would go as follows. First write the problem in this form: \begin{eqnarray*} J &=& \|W^\top W-I\|_F^2 \\ &=& \mathrm{trace}((W^\top W-I)^\top(W^\top W-I))\\ &=& \mathrm{trace}(W^\top WW^\top W - 2W^\top W + I) \end{eqnarray*}

The derivative can now be computed much as for a scalar function. First we split it up into two parts \begin{eqnarray*} J'(W) &=& \frac{\partial}{\partial W}\mathrm{trace}(W^\top WW^\top W - 2W^\top W + I)\\ &=& \frac{\partial}{\partial W}\mathrm{trace}(W^\top WW^\top W) - 2\frac{\partial}{\partial W}\mathrm{trace}(W^\top W)\\ &=& J_1'(W) -2 J_2'(W) \end{eqnarray*} Starting with the simplest term we have \begin{eqnarray*} J_2'(W) &=& \frac{\partial}{\partial W}\mathrm{trace}(W^\top W)\\ &=& \left. \frac{\partial}{\partial W}\mathrm{trace}(W^\top Y + Y^\top W)\right|_{Y=W}\\ &=& \left.Y + (Y^\top)^\top\right|_{Y=W}\\ &=& 2W \end{eqnarray*} And the quartic term: \begin{eqnarray*} J_1'(W) &=& \frac{\partial}{\partial W}\mathrm{trace}(W^\top WW^\top W)\\ &=& \left. \frac{\partial}{\partial W}\mathrm{trace}(W^\top W Y^\top Y + Y^\top Y W^\top W)\right|_{Y=W}\\ &=& 2\left. \frac{\partial}{\partial W}\mathrm{trace}(W^\top W Y^\top Y)\right|_{Y=W}\\ &=& 2\left. \frac{\partial}{\partial W}\mathrm{trace}(W^\top Y Y^\top Y + Y^\top W Y^\top Y )\right|_{Y=W}\\ &=& \left. 2 Y Y^\top Y + 2(Y^\top Y Y^\top ) ^\top\right|_{Y=W}\\ &=& 4 W W^\top W \end{eqnarray*} So the derivative of $J$ should be $$J'(W) = 4 W W^\top W - 4 W = 4 W(W^\top W-I).$$

• Thank you very much for the detailed derivation. Jan 30, 2015 at 11:26

$$\def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\trace#1{\op{Tr}\LR{#1}} \def\qiq{\quad\implies\quad} \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}}$$For typing convenience, define the matrix variable \eqalign{ X &= \LR{W^TW - I} \qiq dX = \LR{W^TdW+dW^TW} \\ } and the Frobenius product (which is a product notation for the trace) \eqalign{ A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{A^TB} \\ A:A &= \|A\|^2_F \qquad \{ {\rm Frobenius\;norm} \} \\ } Write the $$J$$ function using the above notation. Then calculate its differential and gradient. \eqalign{ J &= X:X \\ dJ &= 2X:dX \\&= 2X:\LR{W^TdW+dW^TW} \\&= 2\LR{X+X^T}:W^TdW \\&= 4WX:dW \\ \grad{J}{W} &= 4WX \\ }

Assuming the norm comes from real inner-product. The derivative of the norm can be found as $$\left.\frac{d}{dt}\right|_{0}\|(W+tH)^T(W+tH)-I\|^{2}\\ =\left.\frac{d}{dt}\right|_{0}\langle (W+tH)^T(W+tH)-I,(W+tH)^T(W+tH)-I\rangle\\ =2\langle W^TW-I,H^TW+W^TH\rangle$$

We now use the definition of adjoint corresponding to a simple matrix transpose for real matrices, $\langle Ax,y\rangle=\langle x,A^Ty\rangle$ and similarly for the operators acting from the right; as well as again the symmetry of the real inner product, we find that the two H-linear terms are in fact the same: $$\langle W^TW-I,W^TH\rangle=\langle W(W^TW-I),H\rangle$$ $$\langle W^TW-I,H^TW\rangle=\langle H(W^TW-I),W\rangle\\=\langle H,W(W^TW-I)^T\rangle\\ =\langle H,W(W^TW-I)\rangle\\ =\langle W(W^TW-I),H\rangle$$ Summing up, the derivative is the linear map $$4\langle W(W^TW-I),H\rangle$$ Then, and only because we have the inner product, we can write the gradient vector: $$\frac{\partial J}{\partial W}=4W(W^TW-I)$$