# From derivatives to gradients in backpropagation

Consider the following explanation of backpropagation from Wikipedia:

Given an input–output pair {\displaystyle (x,y)}(x,y), the loss is:

$$C(y,f^{L}(W^{L}f^{L-1}(W^{L-1}\cdots f^{1}(W^{1}x)\cdots )))$$

The derivative of the loss in terms of the inputs is given by the chain rule; note that each term is a total derivative, evaluated at the value of the network (at each node) on the input $$x$$:

$${\frac {dC}{da^{L}}}\cdot {\frac {da^{L}}{dz^{L}}}\cdot {\frac {dz^{L}}{da^{L-1}}}\cdot {\frac {da^{L-1}}{dz^{L-1}}}\cdot {\frac {dz^{L-1}}{da^{L-2}}}\cdots {\frac {da^{1}}{dz^{1}}}\cdot {\frac {\partial z^{1}}{\partial x}}$$

These terms are: the derivative of the loss function; the derivatives of the activation functions; and the matrices of weights:

$$\displaystyle {\frac {dC}{da^{L}}}\cdot (f^{L})'\cdot W^{L}\cdot (f^{L-1})'\cdot W^{L-1}\cdots (f^{1})'\cdot W^{1}.$$

So far so good. Then they say:

The gradient $$\nabla$$ is the transpose of the derivative of the output in terms of the input, so the matrices are transposed and the order of multiplication is reversed, but the entries are the same:

$$\nabla _{x}C=(W^{1})^{T}\cdot (f^{1})'\cdots \cdot (W^{L-1})^{T}\cdot (f^{L-1})'\cdot (W^{L})^{T}\cdot (f^{L})'\cdot \nabla _{a^{L}}C.$$

Why? i.e. why does converting this to "gradient form" (not sure if that's what they actually did above) requires "transposing matrices and reversing the ordering of multiplication"?

In case it helps, I thought this article here (layout conventions in matrix calculus) may help, but I'm not able to cross-reference hat convention was used above, and why vectors / matrices are transposed, and why they chose to re-arrange the ordering of the terms

$${dC \over da^L} \cdot {da^L \over dz^L} \cdots {\partial z^1 \over \partial x}$$
Since every term after $${dC \over da^L}$$ is either a matrix or higher ranked tensor, evaluating it from right to left, as you would, is considerably more expensive than starting with the vector $$dC \over da^L$$ and having it collapse into more simple vector/matrix operations and then transposing it back at the end; hence, we use the fact that for matrices $$A, B, C$$ where $$ABC$$ is defined, $$ABC = (C^T B^T A^T)^T$$:
$$ABC = (C^T B^T A^T)^T = (C^T (AB)^T)^T = ABC$$