I'm looking into convex optimization and am somewhat confused by some concepts of vector calculus. My problem starts by looking at a scalar function: $$J = f(\mathbf y) = f(\mathbf x \mathbf W + \mathbf b)$$

Let's say that I want to calculate $\frac{ \partial J}{\partial \mathbf x}$. My first guess is to split up the question: $$\frac{ \partial J}{\partial \mathbf x} = \frac{ \partial J}{\partial \mathbf y} \frac{ \partial \mathbf y}{\partial \mathbf x}$$

The first half seems easy as it looks like the gradient of $f$. However I'm not sure what $\frac{ \partial \mathbf y}{\partial \mathbf x}$ means. Is this the Jacobian?

If so given that both $\mathbf y$ and $\mathbf x$ are horizontal vectors, I'm not sure if it would be: $$ \begin{bmatrix} \frac{\partial \mathbf y}{\partial x_1} & ... & \frac{\partial \mathbf y}{\partial x_n} \end{bmatrix} $$

Or rather: $$ \begin{bmatrix} \frac{\partial y_1}{\partial \mathbf x} & ... & \frac{\partial y_n}{\partial \mathbf x} \end{bmatrix} $$

Finally, if I wanted to calculate $\frac{ \partial J}{\partial \mathbf W}$ which would seem possible, is there such a thing as $\frac{ \partial \mathbf y}{\partial \mathbf W}$ or $\frac{ \partial \mathbf W}{\partial \mathbf x}$?

  • 1
    $\begingroup$ For your first question, here's something you can try when you get stuck with these types of questions: let $\mathbf x$ be $1\times 1$ and see which one makes sense. For the second question, $\frac {\partial \mathbf y}{\partial \mathbf W}$ could make sense if you allowed $\mathbf W$ to vary. But in this case, it seems like $\mathbf W$ is probably constant and hence it doesn't make sense. $\frac{\partial \mathbf W}{\partial \mathbf x}$ also does not make sense because $\mathbf W$ is not a function of $\mathbf x$. $\endgroup$ – user137731 Jun 5 '17 at 20:11
  • $\begingroup$ If I'm trying to do gradient descent (e.g., this is sort of a neural net with no hidden layers, just one input layer and one output layer with multiple outputs) then if I'm not mistaken $\frac{\partial \mathbf y}{\partial \mathbf W}$ would be needed? $\endgroup$ – wizplum Jun 5 '17 at 20:27
  • $\begingroup$ IDK, but $\frac{\partial \mathbf y}{\partial \mathbf W}$ won't be a matrix -- it'll be a rank 3 tensor. $\endgroup$ – user137731 Jun 5 '17 at 20:29

As you've discovered, it is awkward to apply the chain rule to these types of problems because the intermediate quantities are often higher-order tensors.

A simpler approach is to use differentials. Since $dX$ has exactly the same tensor character as $X$, you can use the familiar rules of scalar/vector/tensor algebra to manipulate it.

Let's being by writing down the variables of interest $$\eqalign{ y &= xW+b \cr J &= f(y) \cr }$$ Now find their differentials $$\eqalign{ dy &= dx\,W \cr\cr dJ &= \frac{\partial f}{\partial y}:dy \cr &= \frac{\partial f}{\partial y}:dx\,W \cr &= \frac{\partial f}{\partial y}W^T:dx \cr\cr \frac{\partial J}{\partial x} &= \frac{\partial f}{\partial y}W^T \cr\cr }$$ In the above, a colon was used to denote the inner/Frobenius product, i.e. $$\eqalign{A:B &= {\rm tr}(A^TB) \cr}$$

  • $\begingroup$ Thanks, this seems like a great approach, but I'm getting a little bit lost in terminology so I may need to take a step back. For starters, what is the reason to use either $ \partial $ or $ d $ in your equations, is there a difference in meaning? Additionally, for the first line in your $ dJ $ derivation, did you start with $ \frac{dJ}{dy} $ and multiply by $ dy$? If so, why does that multiplication turn into a Frobenius product on the right-hand side? $\endgroup$ – wizplum Jun 15 '17 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.