I am studying the Algorithm 6.4 in the textbook Deep Learning, which is about backpropagation.

I am confused by this line:

$$\nabla_{W^{(k)}}J = gh^{(k-1)T}+\lambda\nabla_{W^{(k)}}{\Omega(\theta)}$$

This equation is derived by calculating the gradient of the equation(from Algorithm 6.3) below:

$$a^{(k)}= b^{(k)}+W^{(k)}h^{(k-1)}$$

But shouldn't the gradient of $W^{(k)}h^{(k-1)}$ with respect to $W^{(k)}$ be $h^{(k-1)}$ ?

Why is there a transpose $^T$ here?

Algorithm 6.3 Algorithm 6.4

  • $\begingroup$ I am not sure but if $g$ is a $1 \times n$ vector and $h$ also has the same dimension as $g$ then we need to take the transpose of $h$. $\endgroup$
    – Wolfy
    Oct 25 '18 at 7:51
  • $\begingroup$ Hi Wolfy, thanks for your reply. I think both $g$ and $h$ are vectors, but they don't need to have same dimensions. Because $h^{(k-1)}$ and $g^{(k)}$ are from two neighboring layers in a deep learning model, and these layers may have different width(dimensions). $\endgroup$
    – BioCoder
    Oct 25 '18 at 8:17
  • $\begingroup$ What confused me most is, from mathematical derivation we can not get $gh^{(k-1)T}$. Maybe this is just an arbitrary convention instead of math? $\endgroup$
    – BioCoder
    Oct 25 '18 at 8:26
  • $\begingroup$ @BioCoder It could be, it is a bit unclear to me as well. I am getting interested in machine learning a lot as well. If you want to really understand the mathematics involved in machine learning you should consider studying some more advanced topics in numerical analysis. $\endgroup$
    – Wolfy
    Oct 25 '18 at 18:42

I'm going to use subscripts because they're easier to type, and retains the use of superscripts to indicate things like transposes and conjugates.

Algorithm 6.4 tells you how to calculate the vector $g$. It's a chain of derivatives extending from the output layer back to the $k^{th}$ layer $$\eqalign{ g &= \frac{\partial L}{\partial {\hat y}} \frac{\partial {\hat y}}{\partial h_l} \frac{\partial h_l}{\partial a_l} \frac{\partial a_l}{\partial h_{l-1}} \frac{\partial h_{l-1}}{\partial a_{l-1}} \frac{\partial a_{l-1}}{\partial h_{l-2}} \ldots \frac{\partial h_{k+1}}{\partial a_{k+1}} \frac{\partial a_{k+1}}{\partial h_{k}} \frac{\partial h_{k}}{\partial a_{k}} &= \frac{\partial L}{\partial a_{k}} }$$ even though $\frac{\partial {\hat y}}{\partial h_l}=1,\,\,$ I added it to the chain for clarity.

Use $g$ to write the differential of $L$ then change variables to $W_k$ $$\eqalign{ dL&= g:da_k\cr &= g:dW_k\,h_{k-1}\cr &= gh_{k-1}^T:dW_k\cr \frac{\partial L}{\partial W_k} &= gh_{k-1}^T \cr }$$ where the colon denotes the trace/Frobenius product, i.e. $$A:B = {\rm tr}(A^TB)$$

The properties of the trace allow one to write things like $$\eqalign{ &{\rm tr}(ABC) = {\rm tr}(CAB) = {\rm tr}(BCA) \cr &{\rm tr}(AB) = {\rm tr}(BA) = {\rm tr}(B^TA^T) \cr }$$ which correspond to rules for rearranging the terms in a Frobenius product $$\eqalign{ &A:BC = B^TA:C = AC^T:B \cr &A:B = B:A = B^T:A^T \cr }$$ Note that the object on each side of the colon must have the same shape, i.e. equal numbers of rows and columns. In that sense, it's similar to a Hadamard product.

  • $\begingroup$ It turns out that the Frobenius product is the key point. Thanks! $\endgroup$
    – BioCoder
    Oct 31 '18 at 7:35
  • $\begingroup$ Hello lynn, one question: is there any book/reference paper that denotes the Frobenius Product as a double colon $:$? Thank you $\endgroup$
    – learning
    Mar 23 '19 at 10:12
  • 1
    $\begingroup$ It's a common notation in engineering disciplines like Transport Phenomenon and Continuum Mechanics. Besides the Wikipedia articles, a rather famous textbook is Bird, Steward, and Lightfoot. $\endgroup$
    – lynn
    Mar 23 '19 at 20:36
  • $\begingroup$ @lynn may I ask why dL = g : da_k? I cannot understand why the trace comes into the picture. And why use the differential instead of regular partial derivatives? Thanks a lot :) $\endgroup$
    – Erosennin
    Apr 19 '20 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.