Derivative of Elementwise Function (working on a vector) I have seen an example (it is in terms of neural network back propagation) that I dont understand. 
Given:


*

*$\textbf{a} = \textbf{x}\textbf{W}_{1}+\textbf{b}_{1} $ (where x is dimension (1x5), $W_1$ is (5x3) and $b_1$ is (1x3)) 

*$\textbf{h}=\sigma(\textbf{a})$ is the sigmoid function: $\frac{1}{1+exp(-a_{i})}$ which acts on the n-dimensional vector $a$ element-wise, meaning $\sigma(\textbf{a}) =[\sigma(a_{1}),\sigma(a_{2}),...\sigma(a_{n})]$ 

*$\theta = \textbf{h}\textbf{W}_{2}+\textbf{b}_2$ (where h is dimension (1x3), $W_2$ is (3x5) and $b_2$ is (1x5)) 

*$\hat{\textbf{y}}$= softmax($\theta$) (where $\hat{y}$ is dimension (1x5)) (definition)

*$L=\operatorname{xent}(y, \hat{y})$ (definition)


The derivative of interest is $\frac{\partial L}{\partial x}$ or by the chain rule:
$$\frac{\partial L}{\partial{x}} =\frac{\partial L}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial{\theta}}\frac{\partial{\theta}}{\partial {h}}\frac{\partial{h}}{\partial{a}}\frac{\partial{a}}{\partial{x}}$$
The result they show makes perfect sense to me (almost) 
$((\hat{\textbf{y}}-\textbf{y}) \textbf{W}_{2}^{T})\circ\sigma'(a)\textbf{W}_{1}$
My Questions:


*

*Since $(\hat{\textbf{y}}-\textbf{y})$ is dimension (1x5) they transpose $\textbf{W}_{2}$ to conform to vector matrix multiplication. Is this OK? Can you just transpose a matrix when you want?

*Why the elementwise multiplication by the derivative of $\sigma(a)$ The rationale is that since $\sigma$ is an elementwise operator, this is proper. I dont understand why you would not apply sigma to each element of $\textbf{a}$ and then matrix multiply this result against the vector on the left?

 A: Allow me to restate the problem in terms of column vectors instead of row vectors
$$\eqalign{
a &= W_1^Tx + b_1 &\implies da = W_1^Tdx \cr
h &= \sigma(a) &\implies dh = (H-H^2)\,da,\,\,\,\,&H={\rm Diag}(h) \cr
\theta &= W_2^Th + b_2  &\implies d\theta = W_2^Tdh \cr
y &= {\rm softmax}(\theta)  &\implies dy = (Y-yy^T)\,d\theta,\,\,\,\,&Y={\rm Diag}(y) \cr
L &= -p:\log y &\implies (p,y) \doteq (y,{\hat y}) \cr
}$$
Find the differential of the final (cross entropy) term, and then its gradient
$$\eqalign{
dL
 &= -p:Y^{-1}dy \cr
 &= -p:Y^{-1}(Y-yy^T)d\theta \cr
 &= -p:(I-1y^T)d\theta \cr
 &= (y1^T-I)p:d\theta \cr
 &= (y-p):W_2^Tdh \cr
 &= W_2(y-p):(H-H^2)da \cr
 &= (H-H^2)W_2(y-p):W_1^Tdx \cr
 &= W_1(H-H^2)W_2(y-p):dx \cr
\frac{\partial L}{\partial x} &= W_1(H-H^2)W_2(y-p) \cr
}$$
In some of these steps, I used a colon to denote the trace/Frobenius product
$$A:B = {\rm tr}(A^TB)$$
Casting the final result back into your preferred notation of row vectors and hats and Hadamard products, yields
$$\eqalign{
\frac{\partial L}{\partial x}
 &= \Big(({\hat y}-y)W_2^T\Big)\circ\Big((h-h\circ h)W_1^T\Big) \cr
}$$
