I'm trying to find a derivative of function: $$L = f \cdot y; f = X \cdot W + b$$

Matrices shapes: $X.shape=(1, m), W.shape=(m,10), b.shape=(1, 10), y.shape=(10, 1)$ I'm looking for $\frac{\partial L}{\partial W}$

According to chain-rule: $$\frac{\partial L}{\partial W} = \frac{\partial L}{\partial f} \frac{\partial f}{\partial W} $$

Separately we can find: $$ \frac{\partial L}{\partial f} = y$$ $$ \frac{\partial f}{\partial W} = X$$

And the problem is that the derivative's dimension of $\frac{\partial L}{\partial W} $ according to my formula is $(10, m)$. However, the dimension should coincide with dimension of $W$.

Also I was advised to find differential of $L$:

$$ d(L) = d(f \cdot y) = d(f) \cdot y = d (X \cdot W + b)y = X \cdot dW \cdot y $$ But I do not understand how can I get from this the derivative $\frac{\partial L}{\partial W} $


Let's use a convention where a lowercase Latin letter always represents a column vector, an uppercase Latin is a matrix, and a Greek letter is a scalar.

Using this convention your equations are $$\eqalign{ f &= W^Tx + b \cr \lambda &= f^Ty \cr }$$ As you have noted, the differential of the scalar function is $$\eqalign{ d\lambda &= df^Ty = (dW^Tx)^Ty = x^TdW\,y \cr }$$ Let's develop that a bit further by introducing the Trace function $$\eqalign{ d\lambda &= {\rm Tr}(x^TdW\,y) = {\rm Tr}(yx^TdW) \cr }$$ Then, depending on your preferred Layout Convention, the gradient is either $$\eqalign{ \frac{\partial\lambda}{\partial W} &=yx^T \quad{\rm or}\quad xy^T \cr }$$ Since you expected the the dimensions of the gradient to be those of $W$, it sounds like your preferred layout is $xy^T$

Also note that $\frac{\partial f}{\partial W}\neq X.\,$ The gradient is a 3rd order tensor, while $X$ is just a 2nd order tensor (aka a matrix). The presence of these 3rd and 4th order tensors as intermediate quantities in the chain rule can make it difficult/impossible to use in practice.

The differential approach suggested by your advisor is often simpler because the differential of a matrix is just another matrix quantity, which is easy to handle.

  • $\begingroup$ Thank you very much for your answer, it became more clear for me now! I have 2 questions about your solution: 1) Do I understand correctly that you introduced Trace function, because dλ is scalar so const=Tr(const) ? 2) $\frac{\partial f}{\partial W}$ is 3rd order tensor. Maybe in this case you know how chain rule works in Neural Networks? Because I get derivative from previous layer and I should multiply it by the derivative of current layer according to chain rule. However, as you mentioned, $\frac{\partial f}{\partial W}$ now is a 3rd order tensor, so how can we apply chain rule? $\endgroup$ – Dmitry Denisov Jan 14 at 22:13
  • $\begingroup$ And even if X is matrix then f is also a matrix and we should take a derivative: matrix-by-matrix? $\endgroup$ – Dmitry Denisov Jan 15 at 12:14
  • $\begingroup$ @DmitryDenisov 1) Yes, ${\rm Tr}(scalar)=scalar,\,$ 2) the gradient really is a 3rd order tensor. The point is you never need to calculate 3rd order (vector-by-matrix) or 4th order (matrix-by-matrix) derivatives, and the programs you write will never calculate such quantities either. These online notes are worth a read. $\endgroup$ – greg Jan 15 at 18:04
  • $\begingroup$ In this row nabla_w[-1] = np.dot(delta, activations[-2].transpose()) they set $\frac{\partial L}{\partial W}$ is equal to $X^T \cdot \delta$, so it doesn't seem like chain rule. I.e. in another case they also should calculate the derivative using differential on paper, however they stated that chain rule is a universal approach $\endgroup$ – Dmitry Denisov Jan 16 at 9:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.