2
$\begingroup$

How does one calculate the derivative of a scalar with respect to a matrix using the chain rule where the intermediate variable is a matrix? For example:

$$\frac{\partial L}{\partial \mathbf W} = \frac{\partial L}{\partial \mathbf Y} \frac{\partial \mathbf Y}{\partial \mathbf W}$$

If $\mathbf Y$ were a vector ($\mathbf y$), the chain rule would suggest that we need to sum across all the individual elements of $\mathbf y$, i.e.

$$\frac{\partial L}{\partial \mathbf W} = \sum_i \frac{\partial L}{\partial y_i} \frac{\partial y_i}{\partial \mathbf W} \text{, where $y_i$ is an element of vector $\mathbf y$}$$

Is it OK to assume that the extension of that rule to the case where $\mathbf Y$ is a matrix is as follows?

$$\frac{\partial L}{\partial \mathbf W} = \sum_{i,j} \frac{\partial L}{\partial \mathbf Y_{i,j}} \frac{\partial \mathbf Y_{i,j}}{\partial \mathbf W}\text{, where $\mathbf Y_{i,j}$ is an element of matrix $\mathbf Y$}$$

$\endgroup$
3
  • $\begingroup$ Per en.wikipedia.org/wiki/Matrix_calculus: “ The chain rule ... unfortunately does not apply in matrix-by-scalar derivatives or scalar-by-matrix derivatives...”. I generally find it less troublesome to work out the differential of $L$ and then extract a derivative from that. $\endgroup$
    – amd
    Feb 22, 2019 at 22:03
  • $\begingroup$ To echo the previous comment, the differential approach is much easier. Even if I were to tell you that your summation rule is correct, exactly how do you plan to evaluate $\frac{\partial Y}{\partial W}?\,$ Hint: it's a fourth-order tensor. $\endgroup$
    – greg
    Feb 22, 2019 at 22:32
  • $\begingroup$ Could you please explain in more detail what the differential approach would look like? I'm not quite sure what you mean. @greg, yes the example I gave is used in the backpropagation algorithm of deep neural networks and it would be a fourth-order tensor, but the goal is to avoid explicitly calculating or displaying it and instead break it down into a sum over the elements of the matrix $Y$, the same way we can do for the case where $y$ is a vector. That's the intuition behind the approach in the example, if it is indeed valid from a mathematical standpoint. $\endgroup$
    – Marko J.
    Feb 23, 2019 at 0:02

1 Answer 1

2
$\begingroup$

Here is a concrete example of the differential approach.

Assume $Y$ is a matrix and the cost function is given by $$L=\|Y\|^2_F = Y:Y$$ where the colon is a convenient product notation for the trace, i.e. $$A:B={\rm Tr}(A^TB)$$ Let's further assume that the relationship to the matrix $W$ is $$Y = X^TW$$ Calculate the differential of the cost function, and then its gradient. $$\eqalign{ dL &= 2Y:dY \cr&= 2Y:X^TdW \cr&= 2XY:dW \cr \frac{\partial L}{\partial W} &= 2XY \cr }$$ If you're working with vectors $(w,y)$ instead of matrices, the derivation is basically unchanged. $$\eqalign{ y &= X^Tw \cr L &= \|y\|^2_F = y:y \cr dL &= 2y:dy \cr&= 2y:X^Tdw \cr&= 2Xy:dw \cr \frac{\partial L}{\partial w} &= 2Xy \cr }$$

$\endgroup$
2
  • $\begingroup$ Thank you for the answer. What about the chain rule summation in my original question, is it valid from a purely mathematical perspective (ignoring the fact that $\frac{\partial \mathbf Y}{\partial \mathbf W}$ would be a fourth-order tensor which is hard to display or interpret directly)? $\endgroup$
    – Marko J.
    Feb 23, 2019 at 10:27
  • $\begingroup$ @MarkoJ. Apply your summation rule to the example given here. If it works out to $2XY$, then it's correct. $\endgroup$
    – greg
    Feb 23, 2019 at 15:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .