1
$\begingroup$

I have a computational graph where one of the nodes is a dot product $b$ between a (3, 3) matrix $W$ and a (3, 1) vector $a$ (visualization below). The context is that of a neural network, which is to be trained with backpropagation. The approach I'm following (Stanford, cs 231n) requires to compute a "local gradient" for each node: which will eventually be chained with derivatives flowing back from the rest of the graph. The local gradient should tell us how $b$ changes as we perturb $W$ and $b$.

dot product

I learned from Bendersky how to compute something called a Jacobian: which extends the idea of partial derivative to vector-valued functions. As I mentioned above, both $W$ and $b$ are parameters to be trained, so it seems that we need two different computations.

Computing $\frac{\partial b}{\partial W}$

Jacobian of b w.r.t W

Computing $\frac{\partial b}{\partial a}$

enter image description here

I would appreciate any feedback on my attempt at calculating the Jacobian of dot product. I found some posts on derivative of dot product on math.SE, but all seemed advanced for my level of understanding, so I thought I'd go through a simple example and ask for feedback.

$\endgroup$
3
$\begingroup$

I don't know how deep your mathematical background is, but I hope you can make some sense of my answer. What is your situation: Your function $b$ is a matrix product $Wa$, and you want to compute the derivatives with respect to the matrix $W$ and with respect to the vector $a$. Your second calculation $\frac{\partial b}{\partial a}$ is correct: The Jacobian is just the matrix $W$.

The first part, calculating the derivative with respect to $W$, is a bit more "tricky", as you are no longer viewing your function $b$ going from $\mathbb R^3 \to \mathbb R^3$, but rather from $\mathbb R^{3\times 3}\to \mathbb R^3$ (you're sticking in a matrix and getting out a vector). So writing down the derivative $\displaystyle \frac{\partial{b}}{\partial W}$ can no longer be written in matrix form. What comes into play here is the notion of a "tensor", which you will probably have read about when reading about machine learning subjects. A tensor is softly speaking a generalization of the matrix / vector notation, and you can think about it from a programming perspective as a multidimensional array.

An example: Consider having 100 images, each of which has 50 times 60 resolution where each pixel has some red-color value between 0 and 255. In order to store the color information about all pixels in a meaningful way, you would introduce a $(100,50,60,256)$-tensor, where the $(k,l,m,n)$-entry is the red-value (this is $n$) of the $(l,m)$-th pixel of the $k$-th image. Does that make sense? I hope so. If you come from a programming background you will probably have used something like that.

Now in your case of the Jacobian $\displaystyle \frac{\partial{b}}{\partial a}$ we store the information of the partial derivatives in the same way: it is a $(3,3)$-tensor, where the $(i,j)$-th entry is exactly $\displaystyle \frac{\partial b_i}{\partial a_j}$: This is your matrix $W$ then in the example above, as you have already correctly calculated.

We generalize this notion to $\displaystyle \frac{\partial b}{\partial W}$ accordingly: You put in a $(3,3)$-tensor (in your case $W$) and get out a $(3)$-tensor (your three-dimensional vector $Wa$). Hence $\displaystyle \frac{\partial b}{\partial W}$ is a $(3,3,3)$-tensor, where you have in the $(i,j,k)$-th entry: $\displaystyle \frac{\partial b_i}{\partial W_{jk}}$. I hope you see now why you can no longer note this down in matrix notation: It's more like a cube. Now let's see how $\displaystyle \frac{\partial b_i}{\partial W_{jk}}$ actually looks like. The $i$-th entry of the vector $b$ is given by:

$$b_i = \sum_{l = 1}^{3}W_{il}a_l.$$

If you differentiate this with respect to $W_{jk}$ you see that

$$\frac{\partial b_i}{\partial W_{jk}} = \begin{cases} 0, & \text{if } i \neq j \\ a_k, & \text{otherwise.} \end{cases}$$

Hence your tensor will look quite sparse.

I hope that this helps a bit in understanding backpropagation. Applying the chain rule to "tensor-derivatives" works then exactly the same way as the usual chain rule you might know from multivariate calculus, you just have more axes that you're multiplying and summing over. Multiplications like that are for example implemented in functions like tensordot in the Python library numpy.

$\endgroup$
  • $\begingroup$ Couldn't have hoped for a more helpful, thoughtful, generous explanation. Thank you very much. $\endgroup$ – Readingtao Feb 19 '18 at 3:09
  • $\begingroup$ I came with the exact problem as @Readingtao. I feel like I've almost understood the content. I followed your explanation and figure out derivative of a fot product. The link to Stanford curse notes gives only a small code. I can't figure out how they came to those simple equations in a couple of lines of code, including transforms of inputs etc. Would you like toexpand your answer to cover it, as well? $\endgroup$ – Celdor Jul 31 at 13:14
  • $\begingroup$ Dear @Celdor, if there's a specific question about the article and your understanding, don't hesitate to open a new question. :) $\endgroup$ – johnnycrab Jul 31 at 14:15
  • $\begingroup$ Thanks. The question is about the formulas for dW and dX as I can't understand where they come from even though I read your answer. $\endgroup$ – Celdor Jul 31 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.