The usual way backprop is written for neural networks is "inconsistent" with the chain rule. To show this, let's use a linear neural network y=ABx (for simplicity, think of A and B as $\in R^{n\times n}$) with a quadratic loss $L = 1/2||t-y||^2$, for $t$ the "target".
"Backprop" is usually thought as "backpropagating" the errors $\delta$ up until the layer whose weight we are updating, and multiplying it by the tranpose of the previous layers activation. In the picture below, to update B, this would give us $\frac{\partial L}{\partial B} = \delta_h x^T$.
But this way of writing things is inconsistent with the chain rule notation, because we need to invert the order of the multiplications! (see picture)
Here's my attempt at resolving the issue, using tensor notation. Apart from some terms in the chain rule being swapped in the backprop notation, there is another inconsistency with the dimensions of the derivatives.
$\frac{\partial h}{\partial B}$ is a mapping from $R^{n\times n}$ to $R^n$, which can be written as a (2,1)-tensor, but backprop simplifies it to $x^T$, a (0,1)-tensor. By an (m,n)-tensor, I mean a tensor formed of $m$ vectors and $n$ covectors. This kind of tensor can "eat" up to $n$ vectors, to give back an $(m,0)$ tensor, or be "eaten" by up to $m$ covectors, to give back a $(0,n)$ tensor.
In this notation, with $A$ a (k,l)-tensor and $B$ a (m,n)-tensor, $\frac{\partial A}{\partial B}$ results in a (k+n,l+m)-tensor. This is because it is a map $(R^{m\times n})^* \rightarrow R^{k\times l} \cong R^{n \times m} \rightarrow R^{k\times l} \cong R^{(k+n) \times (l+m)}$. As a simple example, the gradient of the loss wrt y is $\frac{\partial L}{\partial y}$, where L is a (0,0)-tensor (a scalar) and y a (1,0)-tensor (a vector), which results in a (0,1)-tensor (a covector). Notice that this way, the derivative $\frac{\partial L}{\partial y}$ is the dual of the gradient, so we need to take its transpose to get the gradient, as is done in the very last equality of the below image.
Last bit of notation to understand the image, $\frac{\partial h}{\partial B}$ is a (2,1)-tensor, which I wrote as matrices embedded within another matrix (2 "columns" and 1 "row").
I hope my notation is clear. If there is any better way to do this, I would really like to learn it!
