# Strange derivation of gradient with respect to a matrix

I have a scalar-valued function $y$ and a vector-valued function $\textbf{x}$:

$$y = \underset{(1 \times D_h)}{\textbf{w}^T}\underset{(D_h \times 1)}{\textbf{x}}$$ $$\underset{(D_h \times 1)}{\textbf{x}} = \underset{(D_h \times D_z)}{\textbf{B}}\underset{(D_z \times 1)}{\textbf{z}}$$

My problem is to compute $\nabla_{\textbf{B}}y$. These are the steps I understand so far:

$$\nabla_{\textbf{B}}y = \frac{\partial{y}}{\partial\textbf{x}}\frac{\partial\textbf{x}}{\partial\textbf{B}} = \textbf{w}^T\frac{\partial\textbf{x}}{\partial\textbf{B}}$$

$\frac{\partial\textbf{x}}{\partial\textbf{B}}$ seems to be 3-dimensional.

I took a wild leap at finishing this derivation and got:

$$= \underset{(D_z \times 1)}{\textbf{z}}\underset{(1\times D_h)}{\textbf{w}^T}$$

I computed some two-point approximations of the gradient and this solution seems to be working.

My question is, why does this last step work? What identity is this? It seems very strange to me that the $\textbf{z}$ would show up on the left. I'm probably skipping some steps..

For your problem \eqalign{ y &= w:x \cr &= w:Bz \cr &= wz^T:B \cr \cr dy &= wz^T:dB \cr \cr \frac{\partial y}{\partial B} &= wz^T \cr \cr } where colon represents the inner/Frobenius product, which is really just an infix notation for the matrix trace, i.e. $$A:B = {\rm tr}(A^TB)$$ Note that each term in a Frobenius product has the same shape, a feature that it shares with the elementwise (Hadamard) product. In fact, the Frobenius product is just the sum over all the elements in a Hadamard product.