2
$\begingroup$

I'm asked to differentiate

$$\dfrac{C}{2} \sum^{m}_{j' = 1} \| W^{j'} \|^{2}_{2},$$

according to $w^{j'}_{k}$ which is the $k$th weight of the vector of weight of $j'$.

  • It seems that $\| W^{j'} \|^{2}_{2}$ stands for the the $L2$ norm squared. This seems to indicate that we have: $$\| W^{j'} \|^{2}_{2} = w^{2}_1 + w^{2}_2 + ... + w^{2}_k$$
  • Then to my understanding $\sum^{m}_{j' = 1} \| W^{j'} \|^{2}_{2}$ seems to indicate that we have: $$\sum^{m}_{j' = 1} \| W^{j'} \|^{2}_{2} = (w^{2}_1 + w^{2}_2 + ... + w^{2}_k)_{j'=1} + (w^{2}_1 + w^{2}_2 + ... + w^{2}_k)_{j'=2} + ... + (w^{2}_1 + w^{2}_2 + ... + w^{2}_k)_{j'=m}$$

I have absolutely no clue how to derivate this formula. Do I use the sum rule to get this? $$\dfrac{\partial }{\partial w^{j'}_{k}} =\dfrac{C}{2} (2w_k)_{j'=1} + (2w_k)_{j'=2} + ... + (2w_k)_{j'=m}$$

Edit: This is from my machine learning course. This formula represents the regularization term we add to the loss function of an SVM (Support vector machines) in order to minimize the objective function.

Here $W$ is a vector containing scalars $w$.

$\endgroup$
7
  • 1
    $\begingroup$ Differentiate is a better word to use than derivate FWIW. $\endgroup$ Oct 20, 2020 at 16:20
  • $\begingroup$ I think your answer is actually fine : both what you make of the initial formula, and how you differentiate it after that. Just to be sure , could you provide a link of which document you are reading this material from? If it is class notes, then kindly provide photos, anything to give further context. Also mention which course you are taking ,and what is your background. $\endgroup$ Oct 23, 2020 at 4:46
  • $\begingroup$ Please supply the definition of $W$. Is it a vector, matrix, etc? What do you mean by "according to $w_k^{j'}$"? Note that $j'$ is a summation index.. $\endgroup$
    – jack
    Oct 23, 2020 at 7:20
  • $\begingroup$ @jack I've updated the question with more details. $\endgroup$
    – WindBreeze
    Oct 23, 2020 at 16:17
  • 1
    $\begingroup$ @Servaes We have multiple vectors W. So $W^j$ is the jth vector. We have m vectors. $\endgroup$
    – WindBreeze
    Oct 23, 2020 at 19:54

2 Answers 2

2
+100
$\begingroup$

You have $m$ vectors $W^1,\ldots,W^m$, each of length $n$. So all in all you have $mn$ variables $w^j_k$ with $1\leq j\leq m$ and $1\leq k\leq n$. By definition of the $L^2$-norm you have $$\frac{C}{2}\sum_{j=1}^m||W^j||_2^2=\frac{C}{2}\sum_{j=1}^m\sum_{k=1}^n(w^j_k)^2,$$ which is simply a sum of squares. So for any particular variable $w^j_k$ you have $$\frac{\partial}{\partial w^j_k}(w^i_l)^2=\begin{cases}2w^j_k&\text{ if $i=j$ and $l=k$}\\0&\text{ otherwise }\end{cases}$$ Then by linearity of derivatives it quickly follows that $$\frac{\partial}{\partial w^j_k}\frac{C}{2}\sum_{i=1}^m||W^i||_2^2=\frac{C}{2}\sum_{i=1}^m\sum_{l=1}^n\frac{\partial}{\partial w^j_k}(w^i_l)^2=Cw^j_k.$$

$\endgroup$
6
  • $\begingroup$ Why does the power of 2 disappear while doing the sum of all the (w)^2? $\endgroup$
    – WindBreeze
    Oct 23, 2020 at 21:00
  • $\begingroup$ What power of $2$ do you mean? $\endgroup$
    – Servaes
    Oct 23, 2020 at 21:12
  • 1
    $\begingroup$ In the final answer of the derivative where did the 2 in 2w^j_k went?Is Cw inside the \sum or outside? I suspect inside because you kept the indices j and k. $\endgroup$
    – WindBreeze
    Oct 23, 2020 at 22:28
  • $\begingroup$ Ah, I see now. It cancels against the denominator of $\frac{C}{2}$. The rest of the sum vanishes; see the line above. $\endgroup$
    – Servaes
    Oct 23, 2020 at 23:38
  • 1
    $\begingroup$ Thank you now it's clear! It seems that I can only give you the bounty points in 53 mins. $\endgroup$
    – WindBreeze
    Oct 23, 2020 at 23:54
2
$\begingroup$

Use the standard basis vectors $\{e_k\}$ to extract the columns of the matrix, i.e. $$W^j = We_j$$ Let's also use a colon to denote the trace/Frobenius product, i.e. $$A:B \;=\; \sum_{i=1}^m\sum_{j=1}^nA_{ij}B_{ij} \;=\; {\rm Tr}(A^TB)$$ Then the objective function can be expressed in a form which is friendlier to matrix algebra. $$\eqalign{ \phi &= \dfrac{C}{2} \sum^{m}_{j' = 1} \| W^{j'} \|^{2}_{2} \\ &= \tfrac 12C\left(\sum_{j=1}^m We_j:We_j\right) \\ &= \tfrac 12C\left(\sum_{j=1}^m e_je_j^T:W^TW\right) \\ &= \tfrac 12C\Big(I:W^TW\Big) \\ &= \tfrac 12C\,W:W \\ }$$ Now calculating the gradient is easy. $$\eqalign{ d\phi &= \tfrac 12C\,(W:dW+dW:W) \\ &= CW:dW \\ \frac{\partial\phi}{\partial W} &= CW \\ }$$ If you wish to extract individual components, simply pre/post multiply by the basis vectors. $$\eqalign{ e_i^T\left(\frac{\partial\phi}{\partial W}\right)e_j &= Ce_i^TWe_j \\ \frac{\partial\phi}{\partial W_{ij}} &= CW_{ij} \\ }$$

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .