0
$\begingroup$

I have the following equation: $$\begin{align}Wx+b &= O\\ L &=f(O)\end{align}$$

where $W$ is a matrix of $N\times N$, $x, b, O$ are column vector of size $N\times 1$, $L$ is a scalar.

I want to calculate $\frac{\partial{L}}{\partial{W}}$, which will be a matrix of size $N\times N$. According to chain rule, we also have $\frac{\partial{L}}{\partial{W}} = \frac{\partial{L}}{\partial{O}}\frac{\partial{O}}{\partial{W}}$(I am not so sure about this equation). It is easy to calculate $\frac{\partial{L}}{\partial{O}}$, which is of size $N\times 1$. But the $\frac{\partial{O}}{\partial{W}}$ is 3-dimensional matrix of $N\times N \times N$, if I understand it correctly.

My question is, is the chain rule I give right? If it is right, how to multiply 2-dimensional matrix with 3-dimensional matrix? If I am wrong, please correct me.

$\endgroup$
  • $\begingroup$ Of course $\partial L/\partial W$ depends on $f$. You have$$\partial L/\partial W = \nabla f(Wx+b)^T\begin{pmatrix}x^T & 0 & & \\0 & x^T & & \\&&\ddots & \\ & & & x^T\end{pmatrix}.$$Note that the big matrix is of size $N\times N^2$. $\endgroup$ – amsmath Oct 15 '17 at 13:16
1
$\begingroup$

I always feel insecure about these matrix derivatives, so I write everything in terms of components and expand the matrix products: $$ O_{i}= \sum_j W_{ij} \,x_{j} +b_{i} $$ Then, for any indices $k,j$: $$ \frac{\partial O_{i}}{\partial W_{kj} } = \begin{cases} x_j & i=k \\ 0 & i\ne k \end{cases} $$

Now you can compute $\frac{\partial L}{\partial W}$ by applying the chain rule over all components $O_i$ of $f$: $$ \frac{\partial L}{\partial W_{kj} } = \sum_i \frac{\partial L}{\partial O_{i} }\frac{\partial O_{i}}{\partial W_{kj} }=\frac{\partial L}{\partial O_{k} } x_j $$ since the terms with $i\ne k$ vanish.

And that is... but maybe you want to rewrite this expression as a nice matrix product. Then, consider the vector $g$ with $g_k=\frac{\partial L}{\partial O_{k}} $ and the matrix of derivatives $A$ with $A_{kj}=\frac{\partial L}{\partial W_{kj} }$ yielding $A=g\cdot x^\top$ so with some abuse of notation you can write: $$ \frac{\partial L}{\partial W } =g\cdot x^\top $$

$\endgroup$
1
$\begingroup$

In matrix calculus, it is often easier to employ differentials than the chain rule. This is because the intermediate quantities in the chain rule are often 3rd and 4th order tensors, whereas the differential of a matrix is just another matrix.

For ease of typing, let's write the gradient of $L$ (with respect to $O$) as $$g=\frac{\partial L}{\partial O}$$ Then the differential and gradient (with respect to $W$) are $$\eqalign{ dL &= g:dO = g:dWx = gx^T:dW \cr \frac{\partial L}{\partial W} &= gx^T \cr }$$ where a colon denotes the trace/Frobenius product, i.e. $\,\,A:B={\rm tr}(A^TB)$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.