# Is there a formula for differentiating a nonlinear function by a matrix?

I'm struggling with matrix notation for representing the derivative of a nonlinear function by a matrix. Specifically, I'm calculating a gradient. I have:

$\quad \frac{\partial}{\partial \mathbf{W}} \phi ( \mathbf{W} \vec{x} )^T \vec{\beta}$

Where, say, $\vec{x}$ is $n \times 1$, $\vec{\beta}$ is $m \times 1$, and $\mathbf{W}$ is $m \times n$. To simplify the question, let's say $\vec{x}$ and $\vec{\beta}$ are constant vectors.

What has me stuck is $\phi(u)$ - a nonlinear transformation of its argument vector. (For my purpose it is the sigmoid function $\frac{1}{1 + e^{-u}}$). I can calculate this gradient exhaustively, but is there a shortcut that has a clean representation in matrix notation?

For example, if the problem were simply:

$\quad \frac{\partial}{\partial \mathbf{W}} (\mathbf{W} \vec{x})^T \vec{\beta}$

Then I could do this very neatly:

$\quad \frac{\partial}{\partial \mathbf{W}} (\mathbf{W} \vec{x})^T \vec{\beta} = \frac{\partial}{\partial \mathbf{W}} \vec{x}^T \mathbf{W}^T \vec{\beta} = \vec{\beta} \vec{x}^T$

What you need to know is the "trick" for the finding derivative of scalar function applied element-wise to a matrix argument. Assume that you have a scalar function $S(x)$ whose derivative is known to be $S'(x)$. When you apply this element-wise to a matrix, the differential is \eqalign{ dS({\bf X}) &= S'({\bf X})\circ d{\bf X} \cr } where $\circ$ denotes the Hadamard product.
For the Logistic function, the derivative is known to be: $\,\,\,\sigma' = \sigma - \sigma^2$.
\eqalign{ f &= \sigma({\bf Wx})^T{\bf b} \cr &= \sigma^T{\bf b} \cr &= {\bf b}:\sigma \cr\cr df &= {\bf b}:d\sigma \cr &= {\bf b}:\sigma'\circ d({\bf Wx}) \cr &= {\bf b}\circ\sigma':d{\bf W}\,{\bf x} \cr &= ({\bf b}\circ\sigma')\,{\bf x}^T:d{\bf W} \cr &= ({\bf b}\circ\sigma-{\bf b}\circ\sigma\circ\sigma)\,{\bf x}^T:d{\bf W} \cr } Since $df=(\frac{\partial f}{\partial W}:dW),\,$ the gradient is \eqalign{ \frac{\partial f}{\partial {\bf W}} &= ({\bf b}\circ\sigma-{\bf b}\circ\sigma\circ\sigma)\,{\bf x}^T \cr } In the case that the scalar function is the identity function, i.e. $S(x)=x$, then the deriviative is unity $S'(x)=1$.
When applied element-wise to a matrix argument, the result is a matrix of all-ones, which just happens to be the identity element for the Hadamard product. So $(b\circ\sigma')$ would be replaced by $(b\circ 1=b)$ in the differential, yielding a gradient of \eqalign{ \frac{\partial f}{\partial {\bf W}} &= {\bf b}\,{\bf x}^T \cr } which is the result that you already knew.