# Chain rule where intermediate variable is a matrix

How does one calculate the derivative of a scalar with respect to a matrix using the chain rule where the intermediate variable is a matrix? For example:

$$\frac{\partial L}{\partial \mathbf W} = \frac{\partial L}{\partial \mathbf Y} \frac{\partial \mathbf Y}{\partial \mathbf W}$$

If $$\mathbf Y$$ were a vector ($$\mathbf y$$), the chain rule would suggest that we need to sum across all the individual elements of $$\mathbf y$$, i.e.

$$\frac{\partial L}{\partial \mathbf W} = \sum_i \frac{\partial L}{\partial y_i} \frac{\partial y_i}{\partial \mathbf W} \text{, where y_i is an element of vector \mathbf y}$$

Is it OK to assume that the extension of that rule to the case where $$\mathbf Y$$ is a matrix is as follows?

$$\frac{\partial L}{\partial \mathbf W} = \sum_{i,j} \frac{\partial L}{\partial \mathbf Y_{i,j}} \frac{\partial \mathbf Y_{i,j}}{\partial \mathbf W}\text{, where \mathbf Y_{i,j} is an element of matrix \mathbf Y}$$

• Per en.wikipedia.org/wiki/Matrix_calculus: “ The chain rule ... unfortunately does not apply in matrix-by-scalar derivatives or scalar-by-matrix derivatives...”. I generally find it less troublesome to work out the differential of $L$ and then extract a derivative from that.
– amd
Feb 22, 2019 at 22:03
• To echo the previous comment, the differential approach is much easier. Even if I were to tell you that your summation rule is correct, exactly how do you plan to evaluate $\frac{\partial Y}{\partial W}?\,$ Hint: it's a fourth-order tensor.
– greg
Feb 22, 2019 at 22:32
• Could you please explain in more detail what the differential approach would look like? I'm not quite sure what you mean. @greg, yes the example I gave is used in the backpropagation algorithm of deep neural networks and it would be a fourth-order tensor, but the goal is to avoid explicitly calculating or displaying it and instead break it down into a sum over the elements of the matrix $Y$, the same way we can do for the case where $y$ is a vector. That's the intuition behind the approach in the example, if it is indeed valid from a mathematical standpoint. Feb 23, 2019 at 0:02

Assume $$Y$$ is a matrix and the cost function is given by $$L=\|Y\|^2_F = Y:Y$$ where the colon is a convenient product notation for the trace, i.e. $$A:B={\rm Tr}(A^TB)$$ Let's further assume that the relationship to the matrix $$W$$ is $$Y = X^TW$$ Calculate the differential of the cost function, and then its gradient. \eqalign{ dL &= 2Y:dY \cr&= 2Y:X^TdW \cr&= 2XY:dW \cr \frac{\partial L}{\partial W} &= 2XY \cr } If you're working with vectors $$(w,y)$$ instead of matrices, the derivation is basically unchanged. \eqalign{ y &= X^Tw \cr L &= \|y\|^2_F = y:y \cr dL &= 2y:dy \cr&= 2y:X^Tdw \cr&= 2Xy:dw \cr \frac{\partial L}{\partial w} &= 2Xy \cr }
• Thank you for the answer. What about the chain rule summation in my original question, is it valid from a purely mathematical perspective (ignoring the fact that $\frac{\partial \mathbf Y}{\partial \mathbf W}$ would be a fourth-order tensor which is hard to display or interpret directly)? Feb 23, 2019 at 10:27
• @MarkoJ. Apply your summation rule to the example given here. If it works out to $2XY$, then it's correct.