12
$\begingroup$

In linear regression, the loss function is expressed as

$$\frac1N \left\|XW-Y\right\|_{\text{F}}^2$$

where $X, W, Y$ are matrices. Taking derivative w.r.t $W$ yields

$$\frac 2N \, X^T(XW-Y)$$

Why is this so?

$\endgroup$
  • $\begingroup$ Don't be self-deprecating, your question is not dumb :) +1 $\endgroup$ – Andres Mejia Feb 4 '17 at 7:23
13
$\begingroup$

Let

$$\begin{array}{rl} f (\mathrm W) &:= \| \mathrm X \mathrm W - \mathrm Y \|_{\text{F}}^2 = \mbox{tr} \left( (\mathrm X \mathrm W - \mathrm Y)^{\top} (\mathrm X \mathrm W - \mathrm Y) \right)\\ &\,= \mbox{tr} \left( \mathrm W^{\top} \mathrm X^{\top} \mathrm X \mathrm W - \mathrm Y^{\top} \mathrm X \mathrm W - \mathrm W^{\top} \mathrm X^{\top} \mathrm Y + \mathrm Y^{\top} \mathrm Y \right)\end{array}$$

Differentiating with respect to $\mathrm W$,

$$\nabla_{\mathrm W} f (\mathrm W) = 2 \, \mathrm X^{\top} \mathrm X \mathrm W - 2 \, \mathrm X^{\top} \mathrm Y = 2 \, \mathrm X^{\top} \left( \mathrm X \mathrm W - \mathrm Y \right)$$

$\endgroup$
  • 1
    $\begingroup$ I'm just saying what's the derivative of $ \left\| X W - Y \right\|_{F}^{2} $ with respect to $ X $. Just curious. $\endgroup$ – Royi Sep 8 '17 at 15:28
  • 1
    $\begingroup$ I get it is $ 2 \left( X W - Y \right) {W}^{T} $. $\endgroup$ – Royi Sep 8 '17 at 15:30
  • 1
    $\begingroup$ @Royi I just got the exact same. $\endgroup$ – Rodrigo de Azevedo Sep 8 '17 at 15:32
  • 1
    $\begingroup$ @kong If you want a step-by-step derivation, use the directional derivative. $\endgroup$ – Rodrigo de Azevedo Dec 29 '17 at 12:48
  • 1
    $\begingroup$ @kong You can always use the matrix cookbook. $\endgroup$ – Rodrigo de Azevedo Dec 29 '17 at 12:54
7
$\begingroup$

Let $X=(x_{ij})_{ij}$ and similarly for the other matrices. We are trying to differentiate $$ \|XW-Y\|^2=\sum_{i,j}(x_{ik}w_{kj}-y_{ij})^2\qquad (\star) $$ with respect to $W$. The result will be a matrix whose $(i,j)$ entry is the derivative of $(\star)$ with respect to the variable $w_{ij}$.

So think of $(i,j)$ as being fixed now. Only some of the terms in $(\star)$ depend on $w_{ij}$. Taking their derivative gives $$ \frac{d\|XW-Y\|^2}{dw_{ij}}=\sum_{k}2x_{ki}(x_{ki}w_{ij}-y_{kj})=\left[2X^T(XW-Y)\right]_{i,j}. $$

$\endgroup$
  • $\begingroup$ I found your answer very helpful! Thanks so much :). $\endgroup$ – Elchanan Solomon Sep 20 '18 at 10:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.