28
$\begingroup$

In linear regression, the loss function is expressed as

$$\frac1N \left\|XW-Y\right\|_{\text{F}}^2$$

where $X, W, Y$ are matrices. Taking derivative w.r.t $W$ yields

$$\frac 2N \, X^T(XW-Y)$$

Why is this so?

$\endgroup$
0

5 Answers 5

Reset to default
34
$\begingroup$

Let

$$\begin{array}{rl} f (\mathrm W) &:= \| \mathrm X \mathrm W - \mathrm Y \|_{\text{F}}^2 = \mbox{tr} \left( (\mathrm X \mathrm W - \mathrm Y)^{\top} (\mathrm X \mathrm W - \mathrm Y) \right)\\ &\,= \mbox{tr} \left( \mathrm W^{\top} \mathrm X^{\top} \mathrm X \mathrm W - \mathrm Y^{\top} \mathrm X \mathrm W - \mathrm W^{\top} \mathrm X^{\top} \mathrm Y + \mathrm Y^{\top} \mathrm Y \right)\end{array}$$

Differentiating with respect to $\mathrm W$,

$$\nabla_{\mathrm W} f (\mathrm W) = 2 \, \mathrm X^{\top} \mathrm X \mathrm W - 2 \, \mathrm X^{\top} \mathrm Y = \color{blue}{2 \, \mathrm X^{\top} \left( \mathrm X \mathrm W - \mathrm Y \right)}$$

$\endgroup$
17
  • 1
    $\begingroup$ I'm just saying what's the derivative of $ \left\| X W - Y \right\|_{F}^{2} $ with respect to $ X $. Just curious. $\endgroup$
    – Royi
    Sep 8, 2017 at 15:28
  • 4
    $\begingroup$ @kong You can always use the matrix cookbook. $\endgroup$
    – Rodrigo de Azevedo
    Dec 29, 2017 at 12:54
  • 2
    $\begingroup$ @kong The derivatives of the linear terms are easy. Just use the properties of the trace and the definition of the Frobenius inner product. The derivative of the quadratic term is not so easy, but one can use the definition of the directional derivative. $\endgroup$
    – Rodrigo de Azevedo
    Dec 29, 2017 at 13:03
  • 2
    $\begingroup$ @kong Section 2.5.2, equation 108. $\endgroup$
    – Rodrigo de Azevedo
    Dec 29, 2017 at 13:40
  • 2
    $\begingroup$ @kong No, because $(X^T X)^T = X^T X$. When you transpose a product of matrices, the order is reversed. $\endgroup$
    – Rodrigo de Azevedo
    Dec 29, 2017 at 14:17
15
$\begingroup$

Let $X=(x_{ij})_{ij}$ and similarly for the other matrices. We are trying to differentiate $$ \|XW-Y\|^2=\sum_{i,j}(x_{ik}w_{kj}-y_{ij})^2\qquad (\star) $$ with respect to $W$. The result will be a matrix whose $(i,j)$ entry is the derivative of $(\star)$ with respect to the variable $w_{ij}$.

So think of $(i,j)$ as being fixed now. Only some of the terms in $(\star)$ depend on $w_{ij}$. Taking their derivative gives $$ \frac{d\|XW-Y\|^2}{dw_{ij}}=\sum_{k}2x_{ki}(x_{ki}w_{ij}-y_{kj})=\left[2X^T(XW-Y)\right]_{i,j}. $$

$\endgroup$
4
  • 1
    $\begingroup$ I found your answer very helpful! Thanks so much :). $\endgroup$
    – Elchanan Solomon
    Sep 20, 2018 at 10:04
  • $\begingroup$ There's a sum missing in the first expression. It should be: $\sum_{i,j}(\sum_k x_{ik}w_{kj}-y_{ij})^2$ $\endgroup$
    – Maverick Meerkat
    Mar 13, 2021 at 20:58
  • $\begingroup$ ... and also in the 2nd expression. It might be easier to use the same indices: $L = \sum_{i}\sum_j(\sum_k x_{ik}w_{kj}-y_{ij})^2$, $\frac{\partial L}{\partial w_{k'j'}}=\sum_i 2(\sum_k x_{ik}w_{kj'}-y_{ij'})x_{ik'} = 2X^T(XW-Y)$. $\endgroup$
    – Maverick Meerkat
    Mar 13, 2021 at 21:14
  • $\begingroup$ This should be accepted :>. Much more intuitive than using trace and expanding everything out. Thank you! $\endgroup$
    – MathematicsBeginner
    May 17 at 9:11
3
$\begingroup$

Just want to have more details on the process. The process should be Denote $X = [x_{ij}], W = [w_{ij}], Y = [y_{ij}]$, then we have $$ \left \| XW - Y \right \|^{2} = \sum_{k, j} (\sum_{i} x_{ki} w_{ij} - y_{kj})^{2}, $$ This is a scalar and by taking the derivative w.r.t. the matrix $W$ we get a matrix. By taking $i, j$ as the known number, we get $$ \frac{d \left \| XW - Y \right \|^{2}}{d w_{ij}} = \sum_{k} 2x_{ki} (\sum_{i} x_{ki} w_{ij} - y_{kj})\\ = \sum_{k} 2x_{ki} (XW - Y)_{kj} \\ = [2 X^{T} (XW - Y)]_{ij} $$ Thus we have $$ \frac{d \left \| XW - Y \right \|^{2}}{d W} = 2 X^{T} (XW - Y) $$ First time answering a question, hope it is right, thanks!

$\endgroup$
0
$\begingroup$

Roughly speaking, the $\textbf{Jacobian}$ of $f$ at point $x$ is the matrix/tensor $B$ such that we have \begin{equation}f(x+\delta)=f(x) + B\delta+ o(\|\delta\|).\end{equation} So, if $$f(W)=\|XW-Y\|_F^2,$$ then \begin{equation} f(W+\delta)=\|X(W+\delta)-Y\|_F^2=\|XW-Y+X\delta\|_F^2=\|XW-Y\|_F^2+2\langle XW-Y,X\delta \rangle +\|X\delta\|_F^2. \end{equation} Note that we then have \begin{equation} f(W+\delta)=f(W)+2\langle X^T( XW-Y),\delta \rangle +o(\|\delta\|)= f(W)+2\left(X^T( XW-Y)\right)^T\delta +o(\|\delta\|). \end{equation} So, the Jacobian of $f$ is $2\left(X^T( XW-Y)\right)^T$, implying that the gradient is its transpose.

This Taylor expansion idea is a smart trick to make your life easier while taking derivatives.

$\endgroup$
0
$\begingroup$

Let $U(W;X,Y)=XW-Y$, $L(U;N)=\frac{1}{N}U:U$ ("$:$" denote Frobenius inner product)

Using Fréchet derivative:

$L(U+dU)-L(U)=\frac{1}{N}(tr((U+dU)^T(U+dU))-tr((U^TU))=\frac{1}{N}tr(2U^TdU)=\frac{2}{N}U:dU$

$dL=\frac{2}{N}U:dU$

Also, $U(W+dW)-U(W)=XdW$

$dU=XdW$

By chain rule,

$dL=\frac{2}{N}U:(XdW)=tr(\frac{2}{N}U^TXdW)=(\frac{2}{N}X^TU):dW$

$\frac{dL}{dW}=\frac{2}{N}X^TU=\frac{2}{N}X^T(XW-Y)$

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .