Vector-by-vector and vector-by-matrix derivatives

I'm looking into convex optimization and am somewhat confused by some concepts of vector calculus. My problem starts by looking at a scalar function: $$J = f(\mathbf y) = f(\mathbf x \mathbf W + \mathbf b)$$

Let's say that I want to calculate $\frac{ \partial J}{\partial \mathbf x}$. My first guess is to split up the question: $$\frac{ \partial J}{\partial \mathbf x} = \frac{ \partial J}{\partial \mathbf y} \frac{ \partial \mathbf y}{\partial \mathbf x}$$

The first half seems easy as it looks like the gradient of $f$. However I'm not sure what $\frac{ \partial \mathbf y}{\partial \mathbf x}$ means. Is this the Jacobian?

If so given that both $\mathbf y$ and $\mathbf x$ are horizontal vectors, I'm not sure if it would be: $$\begin{bmatrix} \frac{\partial \mathbf y}{\partial x_1} & ... & \frac{\partial \mathbf y}{\partial x_n} \end{bmatrix}$$

Or rather: $$\begin{bmatrix} \frac{\partial y_1}{\partial \mathbf x} & ... & \frac{\partial y_n}{\partial \mathbf x} \end{bmatrix}$$

Finally, if I wanted to calculate $\frac{ \partial J}{\partial \mathbf W}$ which would seem possible, is there such a thing as $\frac{ \partial \mathbf y}{\partial \mathbf W}$ or $\frac{ \partial \mathbf W}{\partial \mathbf x}$?

• For your first question, here's something you can try when you get stuck with these types of questions: let $\mathbf x$ be $1\times 1$ and see which one makes sense. For the second question, $\frac {\partial \mathbf y}{\partial \mathbf W}$ could make sense if you allowed $\mathbf W$ to vary. But in this case, it seems like $\mathbf W$ is probably constant and hence it doesn't make sense. $\frac{\partial \mathbf W}{\partial \mathbf x}$ also does not make sense because $\mathbf W$ is not a function of $\mathbf x$. – user137731 Jun 5 '17 at 20:11
• If I'm trying to do gradient descent (e.g., this is sort of a neural net with no hidden layers, just one input layer and one output layer with multiple outputs) then if I'm not mistaken $\frac{\partial \mathbf y}{\partial \mathbf W}$ would be needed? – wizplum Jun 5 '17 at 20:27
• IDK, but $\frac{\partial \mathbf y}{\partial \mathbf W}$ won't be a matrix -- it'll be a rank 3 tensor. – user137731 Jun 5 '17 at 20:29

A simpler approach is to use differentials. Since $dX$ has exactly the same tensor character as $X$, you can use the familiar rules of scalar/vector/tensor algebra to manipulate it.
Let's being by writing down the variables of interest \eqalign{ y &= xW+b \cr J &= f(y) \cr } Now find their differentials \eqalign{ dy &= dx\,W \cr\cr dJ &= \frac{\partial f}{\partial y}:dy \cr &= \frac{\partial f}{\partial y}:dx\,W \cr &= \frac{\partial f}{\partial y}W^T:dx \cr\cr \frac{\partial J}{\partial x} &= \frac{\partial f}{\partial y}W^T \cr\cr } In the above, a colon was used to denote the inner/Frobenius product, i.e. \eqalign{A:B &= {\rm tr}(A^TB) \cr}
• Thanks, this seems like a great approach, but I'm getting a little bit lost in terminology so I may need to take a step back. For starters, what is the reason to use either $\partial$ or $d$ in your equations, is there a difference in meaning? Additionally, for the first line in your $dJ$ derivation, did you start with $\frac{dJ}{dy}$ and multiply by $dy$? If so, why does that multiplication turn into a Frobenius product on the right-hand side? – wizplum Jun 15 '17 at 18:52