# A matrix calculus problem in backpropagation encountered when studying Deep Learning

I am studying the Algorithm 6.4 in the textbook Deep Learning, which is about backpropagation.

I am confused by this line:

$$\nabla_{W^{(k)}}J = gh^{(k-1)T}+\lambda\nabla_{W^{(k)}}{\Omega(\theta)}$$

This equation is derived by calculating the gradient of the equation(from Algorithm 6.3) below:

$$a^{(k)}= b^{(k)}+W^{(k)}h^{(k-1)}$$

But shouldn't the gradient of $$W^{(k)}h^{(k-1)}$$ with respect to $$W^{(k)}$$ be $$h^{(k-1)}$$ ?

Why is there a transpose $$^T$$ here?

• I am not sure but if $g$ is a $1 \times n$ vector and $h$ also has the same dimension as $g$ then we need to take the transpose of $h$. Oct 25 '18 at 7:51
• Hi Wolfy, thanks for your reply. I think both $g$ and $h$ are vectors, but they don't need to have same dimensions. Because $h^{(k-1)}$ and $g^{(k)}$ are from two neighboring layers in a deep learning model, and these layers may have different width(dimensions). Oct 25 '18 at 8:17
• What confused me most is, from mathematical derivation we can not get $gh^{(k-1)T}$. Maybe this is just an arbitrary convention instead of math? Oct 25 '18 at 8:26
• @BioCoder It could be, it is a bit unclear to me as well. I am getting interested in machine learning a lot as well. If you want to really understand the mathematics involved in machine learning you should consider studying some more advanced topics in numerical analysis. Oct 25 '18 at 18:42

I'm going to use subscripts because they're easier to type, and retains the use of superscripts to indicate things like transposes and conjugates.

Algorithm 6.4 tells you how to calculate the vector $$g$$. It's a chain of derivatives extending from the output layer back to the $$k^{th}$$ layer \eqalign{ g &= \frac{\partial L}{\partial {\hat y}} \frac{\partial {\hat y}}{\partial h_l} \frac{\partial h_l}{\partial a_l} \frac{\partial a_l}{\partial h_{l-1}} \frac{\partial h_{l-1}}{\partial a_{l-1}} \frac{\partial a_{l-1}}{\partial h_{l-2}} \ldots \frac{\partial h_{k+1}}{\partial a_{k+1}} \frac{\partial a_{k+1}}{\partial h_{k}} \frac{\partial h_{k}}{\partial a_{k}} &= \frac{\partial L}{\partial a_{k}} } even though $$\frac{\partial {\hat y}}{\partial h_l}=1,\,\,$$ I added it to the chain for clarity.

Use $$g$$ to write the differential of $$L$$ then change variables to $$W_k$$ \eqalign{ dL&= g:da_k\cr &= g:dW_k\,h_{k-1}\cr &= gh_{k-1}^T:dW_k\cr \frac{\partial L}{\partial W_k} &= gh_{k-1}^T \cr } where the colon denotes the trace/Frobenius product, i.e. $$A:B = {\rm tr}(A^TB)$$

The properties of the trace allow one to write things like \eqalign{ &{\rm tr}(ABC) = {\rm tr}(CAB) = {\rm tr}(BCA) \cr &{\rm tr}(AB) = {\rm tr}(BA) = {\rm tr}(B^TA^T) \cr } which correspond to rules for rearranging the terms in a Frobenius product \eqalign{ &A:BC = B^TA:C = AC^T:B \cr &A:B = B:A = B^T:A^T \cr } Note that the object on each side of the colon must have the same shape, i.e. equal numbers of rows and columns. In that sense, it's similar to a Hadamard product.

• It turns out that the Frobenius product is the key point. Thanks! Oct 31 '18 at 7:35
• Hello lynn, one question: is there any book/reference paper that denotes the Frobenius Product as a double colon $:$? Thank you Mar 23 '19 at 10:12
• It's a common notation in engineering disciplines like Transport Phenomenon and Continuum Mechanics. Besides the Wikipedia articles, a rather famous textbook is Bird, Steward, and Lightfoot.
– lynn
Mar 23 '19 at 20:36
• @lynn may I ask why dL = g : da_k? I cannot understand why the trace comes into the picture. And why use the differential instead of regular partial derivatives? Thanks a lot :) Apr 19 '20 at 16:04