# Derivative of Elementwise Function (working on a vector)

I have seen an example (it is in terms of neural network back propagation) that I dont understand.

Given:

• $\textbf{a} = \textbf{x}\textbf{W}_{1}+\textbf{b}_{1}$ (where x is dimension (1x5), $W_1$ is (5x3) and $b_1$ is (1x3))
• $\textbf{h}=\sigma(\textbf{a})$ is the sigmoid function: $\frac{1}{1+exp(-a_{i})}$ which acts on the n-dimensional vector $a$ element-wise, meaning $\sigma(\textbf{a}) =[\sigma(a_{1}),\sigma(a_{2}),...\sigma(a_{n})]$
• $\theta = \textbf{h}\textbf{W}_{2}+\textbf{b}_2$ (where h is dimension (1x3), $W_2$ is (3x5) and $b_2$ is (1x5))
• $\hat{\textbf{y}}$= softmax($\theta$) (where $\hat{y}$ is dimension (1x5)) (definition)
• $L=\operatorname{xent}(y, \hat{y})$ (definition)

The derivative of interest is $\frac{\partial L}{\partial x}$ or by the chain rule:

$$\frac{\partial L}{\partial{x}} =\frac{\partial L}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial{\theta}}\frac{\partial{\theta}}{\partial {h}}\frac{\partial{h}}{\partial{a}}\frac{\partial{a}}{\partial{x}}$$

The result they show makes perfect sense to me (almost)

$((\hat{\textbf{y}}-\textbf{y}) \textbf{W}_{2}^{T})\circ\sigma'(a)\textbf{W}_{1}$

My Questions:

1. Since $(\hat{\textbf{y}}-\textbf{y})$ is dimension (1x5) they transpose $\textbf{W}_{2}$ to conform to vector matrix multiplication. Is this OK? Can you just transpose a matrix when you want?
2. Why the elementwise multiplication by the derivative of $\sigma(a)$ The rationale is that since $\sigma$ is an elementwise operator, this is proper. I dont understand why you would not apply sigma to each element of $\textbf{a}$ and then matrix multiply this result against the vector on the left?
• I tried to reformat a little the equations. The solution has a ${\bf y}$ which does not appear in the formulas before. What's that? Further, the $\log$ is to be taken elemenwise? Further, what's $\circ$ ? – leonbloy Apr 8 '16 at 23:26
• Hi. $y$ is the actual label. It comes from the definition of softmax and cross entropy. If you fully solve the derivative of -log(y_hat) w.r.t. theta it equals (y_hat - y). That part is not really relevant (that was essentially given as part of a previous derivation). I only included it here because it does lead into the transpose of W_2. The circle symbol denotes element-wise multiplication. – B_Miner Apr 9 '16 at 1:19
• Thank you for reformatting - my latex skills are poor. It took me an hour and I couldnt get the partials right :) – B_Miner Apr 9 '16 at 1:23

Allow me to restate the problem in terms of column vectors instead of row vectors \eqalign{ a &= W_1^Tx + b_1 &\implies da = W_1^Tdx \cr h &= \sigma(a) &\implies dh = (H-H^2)\,da,\,\,\,\,&H={\rm Diag}(h) \cr \theta &= W_2^Th + b_2 &\implies d\theta = W_2^Tdh \cr y &= {\rm softmax}(\theta) &\implies dy = (Y-yy^T)\,d\theta,\,\,\,\,&Y={\rm Diag}(y) \cr L &= -p:\log y &\implies (p,y) \doteq (y,{\hat y}) \cr } Find the differential of the final (cross entropy) term, and then its gradient \eqalign{ dL &= -p:Y^{-1}dy \cr &= -p:Y^{-1}(Y-yy^T)d\theta \cr &= -p:(I-1y^T)d\theta \cr &= (y1^T-I)p:d\theta \cr &= (y-p):W_2^Tdh \cr &= W_2(y-p):(H-H^2)da \cr &= (H-H^2)W_2(y-p):W_1^Tdx \cr &= W_1(H-H^2)W_2(y-p):dx \cr \frac{\partial L}{\partial x} &= W_1(H-H^2)W_2(y-p) \cr } In some of these steps, I used a colon to denote the trace/Frobenius product $$A:B = {\rm tr}(A^TB)$$ Casting the final result back into your preferred notation of row vectors and hats and Hadamard products, yields \eqalign{ \frac{\partial L}{\partial x} &= \Big(({\hat y}-y)W_2^T\Big)\circ\Big((h-h\circ h)W_1^T\Big) \cr }