# Gradient of squared Frobenius norm of a matrix

In linear regression, the loss function is expressed as

$$\frac1N \left\|XW-Y\right\|_{\text{F}}^2$$

where $X, W, Y$ are matrices. Taking derivative w.r.t $W$ yields

$$\frac 2N \, X^T(XW-Y)$$

Why is this so?

## 5 Answers

Let

$$\begin{array}{rl} f (\mathrm W) &:= \| \mathrm X \mathrm W - \mathrm Y \|_{\text{F}}^2 = \mbox{tr} \left( (\mathrm X \mathrm W - \mathrm Y)^{\top} (\mathrm X \mathrm W - \mathrm Y) \right)\\ &\,= \mbox{tr} \left( \mathrm W^{\top} \mathrm X^{\top} \mathrm X \mathrm W - \mathrm Y^{\top} \mathrm X \mathrm W - \mathrm W^{\top} \mathrm X^{\top} \mathrm Y + \mathrm Y^{\top} \mathrm Y \right)\end{array}$$

Differentiating with respect to $$\mathrm W$$,

$$\nabla_{\mathrm W} f (\mathrm W) = 2 \, \mathrm X^{\top} \mathrm X \mathrm W - 2 \, \mathrm X^{\top} \mathrm Y = \color{blue}{2 \, \mathrm X^{\top} \left( \mathrm X \mathrm W - \mathrm Y \right)}$$

• I'm just saying what's the derivative of $\left\| X W - Y \right\|_{F}^{2}$ with respect to $X$. Just curious.
– Royi
Sep 8, 2017 at 15:28
• @kong You can always use the matrix cookbook. Dec 29, 2017 at 12:54
• @kong The derivatives of the linear terms are easy. Just use the properties of the trace and the definition of the Frobenius inner product. The derivative of the quadratic term is not so easy, but one can use the definition of the directional derivative. Dec 29, 2017 at 13:03
• @kong Section 2.5.2, equation 108. Dec 29, 2017 at 13:40
• @kong No, because $(X^T X)^T = X^T X$. When you transpose a product of matrices, the order is reversed. Dec 29, 2017 at 14:17

Let $X=(x_{ij})_{ij}$ and similarly for the other matrices. We are trying to differentiate $$\|XW-Y\|^2=\sum_{i,j}(x_{ik}w_{kj}-y_{ij})^2\qquad (\star)$$ with respect to $W$. The result will be a matrix whose $(i,j)$ entry is the derivative of $(\star)$ with respect to the variable $w_{ij}$.

So think of $(i,j)$ as being fixed now. Only some of the terms in $(\star)$ depend on $w_{ij}$. Taking their derivative gives $$\frac{d\|XW-Y\|^2}{dw_{ij}}=\sum_{k}2x_{ki}(x_{ki}w_{ij}-y_{kj})=\left[2X^T(XW-Y)\right]_{i,j}.$$

• I found your answer very helpful! Thanks so much :). Sep 20, 2018 at 10:04
• There's a sum missing in the first expression. It should be: $\sum_{i,j}(\sum_k x_{ik}w_{kj}-y_{ij})^2$ Mar 13, 2021 at 20:58
• ... and also in the 2nd expression. It might be easier to use the same indices: $L = \sum_{i}\sum_j(\sum_k x_{ik}w_{kj}-y_{ij})^2$, $\frac{\partial L}{\partial w_{k'j'}}=\sum_i 2(\sum_k x_{ik}w_{kj'}-y_{ij'})x_{ik'} = 2X^T(XW-Y)$. Mar 13, 2021 at 21:14
• This should be accepted :>. Much more intuitive than using trace and expanding everything out. Thank you! May 17 at 9:11

Just want to have more details on the process. The process should be Denote $$X = [x_{ij}], W = [w_{ij}], Y = [y_{ij}]$$, then we have $$\left \| XW - Y \right \|^{2} = \sum_{k, j} (\sum_{i} x_{ki} w_{ij} - y_{kj})^{2},$$ This is a scalar and by taking the derivative w.r.t. the matrix $$W$$ we get a matrix. By taking $$i, j$$ as the known number, we get $$\frac{d \left \| XW - Y \right \|^{2}}{d w_{ij}} = \sum_{k} 2x_{ki} (\sum_{i} x_{ki} w_{ij} - y_{kj})\\ = \sum_{k} 2x_{ki} (XW - Y)_{kj} \\ = [2 X^{T} (XW - Y)]_{ij}$$ Thus we have $$\frac{d \left \| XW - Y \right \|^{2}}{d W} = 2 X^{T} (XW - Y)$$ First time answering a question, hope it is right, thanks!

Roughly speaking, the $$\textbf{Jacobian}$$ of $$f$$ at point $$x$$ is the matrix/tensor $$B$$ such that we have $$$$f(x+\delta)=f(x) + B\delta+ o(\|\delta\|).$$$$ So, if $$f(W)=\|XW-Y\|_F^2,$$ then $$$$f(W+\delta)=\|X(W+\delta)-Y\|_F^2=\|XW-Y+X\delta\|_F^2=\|XW-Y\|_F^2+2\langle XW-Y,X\delta \rangle +\|X\delta\|_F^2.$$$$ Note that we then have $$$$f(W+\delta)=f(W)+2\langle X^T( XW-Y),\delta \rangle +o(\|\delta\|)= f(W)+2\left(X^T( XW-Y)\right)^T\delta +o(\|\delta\|).$$$$ So, the Jacobian of $$f$$ is $$2\left(X^T( XW-Y)\right)^T$$, implying that the gradient is its transpose.

This Taylor expansion idea is a smart trick to make your life easier while taking derivatives.

Let $$U(W;X,Y)=XW-Y$$, $$L(U;N)=\frac{1}{N}U:U$$ ("$$:$$" denote Frobenius inner product)

Using Fréchet derivative:

$$L(U+dU)-L(U)=\frac{1}{N}(tr((U+dU)^T(U+dU))-tr((U^TU))=\frac{1}{N}tr(2U^TdU)=\frac{2}{N}U:dU$$

$$dL=\frac{2}{N}U:dU$$

Also, $$U(W+dW)-U(W)=XdW$$

$$dU=XdW$$

By chain rule,

$$dL=\frac{2}{N}U:(XdW)=tr(\frac{2}{N}U^TXdW)=(\frac{2}{N}X^TU):dW$$

$$\frac{dL}{dW}=\frac{2}{N}X^TU=\frac{2}{N}X^T(XW-Y)$$