I'm struggling with matrix notation for representing the derivative of a nonlinear function by a matrix. Specifically, I'm calculating a gradient. I have:

$\quad \frac{\partial}{\partial \mathbf{W}} \phi ( \mathbf{W} \vec{x} )^T \vec{\beta}$

Where, say, $\vec{x}$ is $n \times 1$, $\vec{\beta}$ is $m \times 1$, and $\mathbf{W}$ is $m \times n$. To simplify the question, let's say $\vec{x}$ and $\vec{\beta}$ are constant vectors.

What has me stuck is $\phi(u)$ - a nonlinear transformation of its argument vector. (For my purpose it is the sigmoid function $\frac{1}{1 + e^{-u}}$). I can calculate this gradient exhaustively, but is there a shortcut that has a clean representation in matrix notation?

For example, if the problem were simply:

$\quad \frac{\partial}{\partial \mathbf{W}} (\mathbf{W} \vec{x})^T \vec{\beta}$

Then I could do this very neatly:

$\quad \frac{\partial}{\partial \mathbf{W}} (\mathbf{W} \vec{x})^T \vec{\beta} = \frac{\partial}{\partial \mathbf{W}} \vec{x}^T \mathbf{W}^T \vec{\beta} = \vec{\beta} \vec{x}^T$


What you need to know is the "trick" for the finding derivative of scalar function applied element-wise to a matrix argument. Assume that you have a scalar function $S(x)$ whose derivative is known to be $S'(x)$. When you apply this element-wise to a matrix, the differential is $$\eqalign{ dS({\bf X}) &= S'({\bf X})\circ d{\bf X} \cr }$$ where $\circ$ denotes the Hadamard product.

For the Logistic function, the derivative is known to be: $\,\,\,\sigma' = \sigma - \sigma^2$.

Now let's rewrite your objective in terms of the Logistic function and the Frobenius product (denoted by a colon), then find its differential
$$\eqalign{ f &= \sigma({\bf Wx})^T{\bf b} \cr &= \sigma^T{\bf b} \cr &= {\bf b}:\sigma \cr\cr df &= {\bf b}:d\sigma \cr &= {\bf b}:\sigma'\circ d({\bf Wx}) \cr &= {\bf b}\circ\sigma':d{\bf W}\,{\bf x} \cr &= ({\bf b}\circ\sigma')\,{\bf x}^T:d{\bf W} \cr &= ({\bf b}\circ\sigma-{\bf b}\circ\sigma\circ\sigma)\,{\bf x}^T:d{\bf W} \cr }$$ Since $df=(\frac{\partial f}{\partial W}:dW),\,$ the gradient is $$\eqalign{ \frac{\partial f}{\partial {\bf W}} &= ({\bf b}\circ\sigma-{\bf b}\circ\sigma\circ\sigma)\,{\bf x}^T \cr }$$ In the case that the scalar function is the identity function, i.e. $S(x)=x$, then the deriviative is unity $S'(x)=1$.

When applied element-wise to a matrix argument, the result is a matrix of all-ones, which just happens to be the identity element for the Hadamard product. So $(b\circ\sigma')$ would be replaced by $(b\circ 1=b)$ in the differential, yielding a gradient of $$\eqalign{ \frac{\partial f}{\partial {\bf W}} &= {\bf b}\,{\bf x}^T \cr }$$ which is the result that you already knew.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.