27
$\begingroup$

In linear regression, the loss function is expressed as

$$\frac1N \left\|XW-Y\right\|_{\text{F}}^2$$

where $X, W, Y$ are matrices. Taking derivative w.r.t $W$ yields

$$\frac 2N \, X^T(XW-Y)$$

Why is this so?

$\endgroup$
0

4 Answers 4

30
$\begingroup$

Let

$$\begin{array}{rl} f (\mathrm W) &:= \| \mathrm X \mathrm W - \mathrm Y \|_{\text{F}}^2 = \mbox{tr} \left( (\mathrm X \mathrm W - \mathrm Y)^{\top} (\mathrm X \mathrm W - \mathrm Y) \right)\\ &\,= \mbox{tr} \left( \mathrm W^{\top} \mathrm X^{\top} \mathrm X \mathrm W - \mathrm Y^{\top} \mathrm X \mathrm W - \mathrm W^{\top} \mathrm X^{\top} \mathrm Y + \mathrm Y^{\top} \mathrm Y \right)\end{array}$$

Differentiating with respect to $\mathrm W$,

$$\nabla_{\mathrm W} f (\mathrm W) = 2 \, \mathrm X^{\top} \mathrm X \mathrm W - 2 \, \mathrm X^{\top} \mathrm Y = \color{blue}{2 \, \mathrm X^{\top} \left( \mathrm X \mathrm W - \mathrm Y \right)}$$


$\endgroup$
17
  • 1
    $\begingroup$ I'm just saying what's the derivative of $ \left\| X W - Y \right\|_{F}^{2} $ with respect to $ X $. Just curious. $\endgroup$
    – Royi
    Sep 8, 2017 at 15:28
  • 3
    $\begingroup$ @kong You can always use the matrix cookbook. $\endgroup$ Dec 29, 2017 at 12:54
  • 2
    $\begingroup$ @kong The derivatives of the linear terms are easy. Just use the properties of the trace and the definition of the Frobenius inner product. The derivative of the quadratic term is not so easy, but one can use the definition of the directional derivative. $\endgroup$ Dec 29, 2017 at 13:03
  • 2
    $\begingroup$ @kong Section 2.5.2, equation 108. $\endgroup$ Dec 29, 2017 at 13:40
  • 2
    $\begingroup$ @kong No, because $(X^T X)^T = X^T X$. When you transpose a product of matrices, the order is reversed. $\endgroup$ Dec 29, 2017 at 14:17
13
$\begingroup$

Let $X=(x_{ij})_{ij}$ and similarly for the other matrices. We are trying to differentiate $$ \|XW-Y\|^2=\sum_{i,j}(x_{ik}w_{kj}-y_{ij})^2\qquad (\star) $$ with respect to $W$. The result will be a matrix whose $(i,j)$ entry is the derivative of $(\star)$ with respect to the variable $w_{ij}$.

So think of $(i,j)$ as being fixed now. Only some of the terms in $(\star)$ depend on $w_{ij}$. Taking their derivative gives $$ \frac{d\|XW-Y\|^2}{dw_{ij}}=\sum_{k}2x_{ki}(x_{ki}w_{ij}-y_{kj})=\left[2X^T(XW-Y)\right]_{i,j}. $$

$\endgroup$
3
  • $\begingroup$ I found your answer very helpful! Thanks so much :). $\endgroup$ Sep 20, 2018 at 10:04
  • $\begingroup$ There's a sum missing in the first expression. It should be: $\sum_{i,j}(\sum_k x_{ik}w_{kj}-y_{ij})^2$ $\endgroup$ Mar 13, 2021 at 20:58
  • $\begingroup$ ... and also in the 2nd expression. It might be easier to use the same indices: $L = \sum_{i}\sum_j(\sum_k x_{ik}w_{kj}-y_{ij})^2$, $\frac{\partial L}{\partial w_{k'j'}}=\sum_i 2(\sum_k x_{ik}w_{kj'}-y_{ij'})x_{ik'} = 2X^T(XW-Y)$. $\endgroup$ Mar 13, 2021 at 21:14
3
$\begingroup$

Just want to have more details on the process. The process should be Denote $X = [x_{ij}], W = [w_{ij}], Y = [y_{ij}]$, then we have $$ \left \| XW - Y \right \|^{2} = \sum_{k, j} (\sum_{i} x_{ki} w_{ij} - y_{kj})^{2}, $$ This is a scalar and by taking the derivative w.r.t. the matrix $W$ we get a matrix. By taking $i, j$ as the known number, we get $$ \frac{d \left \| XW - Y \right \|^{2}}{d w_{ij}} = \sum_{k} 2x_{ki} (\sum_{i} x_{ki} w_{ij} - y_{kj})\\ = \sum_{k} 2x_{ki} (XW - Y)_{kj} \\ = [2 X^{T} (XW - Y)]_{ij} $$ Thus we have $$ \frac{d \left \| XW - Y \right \|^{2}}{d W} = 2 X^{T} (XW - Y) $$ First time answering a question, hope it is right, thanks!

$\endgroup$
0
$\begingroup$

Roughly speaking, the $\textbf{Jacobian}$ of $f$ at point $x$ is the matrix/tensor $B$ such that we have \begin{equation}f(x+\delta)=f(x) + B\delta+ o(\|\delta\|).\end{equation} So, if $$f(W)=\|XW-Y\|_F^2,$$ then \begin{equation} f(W+\delta)=\|X(W+\delta)-Y\|_F^2=\|XW-Y+X\delta\|_F^2=\|XW-Y\|_F^2+2\langle XW-Y,X\delta \rangle +\|X\delta\|_F^2. \end{equation} Note that we then have \begin{equation} f(W+\delta)=f(W)+2\langle X^T( XW-Y),\delta \rangle +o(\|\delta\|)= f(W)+2\left(X^T( XW-Y)\right)^T\delta +o(\|\delta\|). \end{equation} So, the Jacobian of $f$ is $2\left(X^T( XW-Y)\right)^T$, implying that the gradient is its transpose.

This Taylor expansion idea is a smart trick to make your life easier while taking derivatives.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.