Consider the following explanation of backpropagation from Wikipedia:

Given an input–output pair {\displaystyle (x,y)}(x,y), the loss is:

$ C(y,f^{L}(W^{L}f^{L-1}(W^{L-1}\cdots f^{1}(W^{1}x)\cdots )))$

The derivative of the loss in terms of the inputs is given by the chain rule; note that each term is a total derivative, evaluated at the value of the network (at each node) on the input $x$:

${\frac {dC}{da^{L}}}\cdot {\frac {da^{L}}{dz^{L}}}\cdot {\frac {dz^{L}}{da^{L-1}}}\cdot {\frac {da^{L-1}}{dz^{L-1}}}\cdot {\frac {dz^{L-1}}{da^{L-2}}}\cdots {\frac {da^{1}}{dz^{1}}}\cdot {\frac {\partial z^{1}}{\partial x}}$

These terms are: the derivative of the loss function; the derivatives of the activation functions; and the matrices of weights:

$\displaystyle {\frac {dC}{da^{L}}}\cdot (f^{L})'\cdot W^{L}\cdot (f^{L-1})'\cdot W^{L-1}\cdots (f^{1})'\cdot W^{1}.$

So far so good. Then they say:

The gradient $\nabla$ is the transpose of the derivative of the output in terms of the input, so the matrices are transposed and the order of multiplication is reversed, but the entries are the same:

$ \nabla _{x}C=(W^{1})^{T}\cdot (f^{1})'\cdots \cdot (W^{L-1})^{T}\cdot (f^{L-1})'\cdot (W^{L})^{T}\cdot (f^{L})'\cdot \nabla _{a^{L}}C.$

Why? i.e. why does converting this to "gradient form" (not sure if that's what they actually did above) requires "transposing matrices and reversing the ordering of multiplication"?

In case it helps, I thought this article here (layout conventions in matrix calculus) may help, but I'm not able to cross-reference hat convention was used above, and why vectors / matrices are transposed, and why they chose to re-arrange the ordering of the terms


1 Answer 1


This is a quite late, but I also had the same issue while trying to understand back-propagation and implementing it in code. I figure others may run into this as well.

I believe the transpose is there simply to reduce the computational load required to evaluate the first expression, i.e:

$$ {dC \over da^L} \cdot {da^L \over dz^L} \cdots {\partial z^1 \over \partial x} $$

Since every term after ${dC \over da^L}$ is either a matrix or higher ranked tensor, evaluating it from right to left, as you would, is considerably more expensive than starting with the vector $dC \over da^L$ and having it collapse into more simple vector/matrix operations and then transposing it back at the end; hence, we use the fact that for matrices $A, B, C$ where $ABC$ is defined, $ABC = (C^T B^T A^T)^T$:

$$ ABC = (C^T B^T A^T)^T = (C^T (AB)^T)^T = ABC $$

And this makes sense because the gradient of a scalar function is, of course, a vector, and for all practical purposes, the transpose of it is insignificant.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .