# Matrix derivatives, problem with dimensions

I'm trying to find a derivative of function: $$L = f \cdot y; f = X \cdot W + b$$

Matrices shapes: $$X.shape=(1, m), W.shape=(m,10), b.shape=(1, 10), y.shape=(10, 1)$$ I'm looking for $$\frac{\partial L}{\partial W}$$

According to chain-rule: $$\frac{\partial L}{\partial W} = \frac{\partial L}{\partial f} \frac{\partial f}{\partial W}$$

Separately we can find: $$\frac{\partial L}{\partial f} = y$$ $$\frac{\partial f}{\partial W} = X$$

And the problem is that the derivative's dimension of $$\frac{\partial L}{\partial W}$$ according to my formula is $$(10, m)$$. However, the dimension should coincide with dimension of $$W$$.

Also I was advised to find differential of $$L$$:

$$d(L) = d(f \cdot y) = d(f) \cdot y = d (X \cdot W + b)y = X \cdot dW \cdot y$$ But I do not understand how can I get from this the derivative $$\frac{\partial L}{\partial W}$$

Using this convention your equations are \eqalign{ f &= W^Tx + b \cr \lambda &= f^Ty \cr } As you have noted, the differential of the scalar function is \eqalign{ d\lambda &= df^Ty = (dW^Tx)^Ty = x^TdW\,y \cr } Let's develop that a bit further by introducing the Trace function \eqalign{ d\lambda &= {\rm Tr}(x^TdW\,y) = {\rm Tr}(yx^TdW) \cr } Then, depending on your preferred Layout Convention, the gradient is either \eqalign{ \frac{\partial\lambda}{\partial W} &=yx^T \quad{\rm or}\quad xy^T \cr } Since you expected the the dimensions of the gradient to be those of $$W$$, it sounds like your preferred layout is $$xy^T$$
Also note that $$\frac{\partial f}{\partial W}\neq X.\,$$ The gradient is a 3rd order tensor, while $$X$$ is just a 2nd order tensor (aka a matrix). The presence of these 3rd and 4th order tensors as intermediate quantities in the chain rule can make it difficult/impossible to use in practice.
• Thank you very much for your answer, it became more clear for me now! I have 2 questions about your solution: 1) Do I understand correctly that you introduced Trace function, because dλ is scalar so const=Tr(const) ? 2) $\frac{\partial f}{\partial W}$ is 3rd order tensor. Maybe in this case you know how chain rule works in Neural Networks? Because I get derivative from previous layer and I should multiply it by the derivative of current layer according to chain rule. However, as you mentioned, $\frac{\partial f}{\partial W}$ now is a 3rd order tensor, so how can we apply chain rule? – Dmitry Denisov Jan 14 at 22:13
• @DmitryDenisov 1) Yes, ${\rm Tr}(scalar)=scalar,\,$ 2) the gradient really is a 3rd order tensor. The point is you never need to calculate 3rd order (vector-by-matrix) or 4th order (matrix-by-matrix) derivatives, and the programs you write will never calculate such quantities either. These online notes are worth a read. – greg Jan 15 at 18:04
• In this row nabla_w[-1] = np.dot(delta, activations[-2].transpose()) they set $\frac{\partial L}{\partial W}$ is equal to $X^T \cdot \delta$, so it doesn't seem like chain rule. I.e. in another case they also should calculate the derivative using differential on paper, however they stated that chain rule is a universal approach – Dmitry Denisov Jan 16 at 9:57