Derivative of Elementwise Function (working on a vector)

I have seen an example (it is in terms of neural network back propagation) that I don’t understand.

Given:

• $$\mathbf{a} = \mathbf{x}\mathbf{W}_{1}+\mathbf{b}_{1}$$ (where $$\mathbf{x}$$ is dimension ($$1\times5$$), $$\mathbf{W}_1$$ is ($$5\times3$$)) and $$\mathbf{b}_1$$ is ($$1\times3$$))
• $$\textbf{h}=\sigma(\textbf{a})$$ is the sigmoid function: $$\frac{1}{1+\exp(-a_{i})}$$ which acts on the $$n$$-dimensional vector $$a$$ element-wise, meaning $$\sigma(\mathbf{a}) =[\sigma(a_{1}),\sigma(a_{2}),\ldots,\sigma(a_{n})]$$
• $$\theta = \mathbf{h}\mathbf{W}_{2}+\mathbf{b}_2$$ (where $$\mathbf{h}$$ is dimension ($$1\times3$$), $$\mathbf{W}_2$$ is ($$3\times5$$) and $$\mathbf{b}_2$$ is ($$1\times5$$))
• $$\hat{\mathbf{y}}= \operatorname{softmax}(\theta)$$ (where $$\hat{\mathbf{y}}$$ is dimension ($$1\times5$$)) (definition)
• $$L=\operatorname{xent}(y, \hat{y})$$ (definition)

The derivative of interest is $$\frac{\partial L}{\partial x}$$ or by the chain rule:

$$\frac{\partial L}{\partial{x}} =\frac{\partial L}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial{\theta}}\frac{\partial{\theta}}{\partial {h}}\frac{\partial{h}}{\partial{a}}\frac{\partial{a}}{\partial{x}}$$

The result they show makes perfect sense to me (almost)

$$((\hat{\mathbf{y}}-\mathbf{y}) \mathbf{W}_{2}^{T})\circ\sigma'(a)\mathbf{W}_{1}$$

My Questions:

1. Since $$(\hat{\mathbf{y}}-\mathbf{y})$$ is dimension ($$1\times5$$) they transpose $$\mathbf{W}_{2}$$ to conform to vector matrix multiplication. Is this OK? Can you just transpose a matrix when you want?
2. Why the elementwise multiplication by the derivative of $$\sigma(a)$$ The rationale is that since $$\sigma$$ is an elementwise operator, this is proper. I don’t understand why you would not apply sigma to each element of $$\mathbf{a}$$ and then matrix-multiply this result against the vector on the left?
• I tried to reformat a little the equations. The solution has a ${\bf y}$ which does not appear in the formulas before. What's that? Further, the $\log$ is to be taken elemenwise? Further, what's $\circ$ ? Apr 8, 2016 at 23:26
• Hi. $y$ is the actual label. It comes from the definition of softmax and cross entropy. If you fully solve the derivative of -log(y_hat) w.r.t. theta it equals (y_hat - y). That part is not really relevant (that was essentially given as part of a previous derivation). I only included it here because it does lead into the transpose of W_2. The circle symbol denotes element-wise multiplication. Apr 9, 2016 at 1:19
• Thank you for reformatting - my latex skills are poor. It took me an hour and I couldnt get the partials right :) Apr 9, 2016 at 1:23

Allow me to restate the problem in terms of column vectors instead of row vectors \eqalign{ a &= W_1^Tx + b_1 &\implies da = W_1^Tdx \cr h &= \sigma(a) &\implies dh = (H-H^2)\,da,\,\,\,\,&H={\rm Diag}(h) \cr \theta &= W_2^Th + b_2 &\implies d\theta = W_2^Tdh \cr y &= {\rm softmax}(\theta) &\implies dy = (Y-yy^T)\,d\theta,\,\,\,\,&Y={\rm Diag}(y) \cr L &= -p:\log y &\implies (p,y) \doteq (y,{\hat y}) \cr } Find the differential of the final (cross entropy) term, and then its gradient \eqalign{ dL &= -p:Y^{-1}dy \cr &= -p:Y^{-1}(Y-yy^T)d\theta \cr &= -p:(I-1y^T)d\theta \cr &= (y1^T-I)p:d\theta \cr &= (y-p):W_2^Tdh \cr &= W_2(y-p):(H-H^2)da \cr &= (H-H^2)W_2(y-p):W_1^Tdx \cr &= W_1(H-H^2)W_2(y-p):dx \cr \frac{\partial L}{\partial x} &= W_1(H-H^2)W_2(y-p) \cr } In some of these steps, I used a colon to denote the trace/Frobenius product $$A:B = {\rm tr}(A^TB)$$ Casting the final result back into your preferred notation of row vectors and hats and Hadamard products, yields \eqalign{ \frac{\partial L}{\partial x} &= \Big(({\hat y}-y)W_2^T\Big)\circ\Big((h-h\circ h)W_1^T\Big) \cr }