I have seen an example (it is in terms of neural network back propagation) that I dont understand.


  • $\textbf{a} = \textbf{x}\textbf{W}_{1}+\textbf{b}_{1} $ (where x is dimension (1x5), $W_1$ is (5x3) and $b_1$ is (1x3))
  • $\textbf{h}=\sigma(\textbf{a})$ is the sigmoid function: $\frac{1}{1+exp(-a_{i})}$ which acts on the n-dimensional vector $a$ element-wise, meaning $\sigma(\textbf{a}) =[\sigma(a_{1}),\sigma(a_{2}),...\sigma(a_{n})]$
  • $\theta = \textbf{h}\textbf{W}_{2}+\textbf{b}_2$ (where h is dimension (1x3), $W_2$ is (3x5) and $b_2$ is (1x5))
  • $\hat{\textbf{y}}$= softmax($\theta$) (where $\hat{y}$ is dimension (1x5)) (definition)
  • $L=\operatorname{xent}(y, \hat{y})$ (definition)

The derivative of interest is $\frac{\partial L}{\partial x}$ or by the chain rule:

$$\frac{\partial L}{\partial{x}} =\frac{\partial L}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial{\theta}}\frac{\partial{\theta}}{\partial {h}}\frac{\partial{h}}{\partial{a}}\frac{\partial{a}}{\partial{x}}$$

The result they show makes perfect sense to me (almost)

$((\hat{\textbf{y}}-\textbf{y}) \textbf{W}_{2}^{T})\circ\sigma'(a)\textbf{W}_{1}$

My Questions:

  1. Since $(\hat{\textbf{y}}-\textbf{y})$ is dimension (1x5) they transpose $\textbf{W}_{2}$ to conform to vector matrix multiplication. Is this OK? Can you just transpose a matrix when you want?
  2. Why the elementwise multiplication by the derivative of $\sigma(a)$ The rationale is that since $\sigma$ is an elementwise operator, this is proper. I dont understand why you would not apply sigma to each element of $\textbf{a}$ and then matrix multiply this result against the vector on the left?
  • $\begingroup$ I tried to reformat a little the equations. The solution has a ${\bf y}$ which does not appear in the formulas before. What's that? Further, the $\log$ is to be taken elemenwise? Further, what's $\circ$ ? $\endgroup$ – leonbloy Apr 8 '16 at 23:26
  • $\begingroup$ Hi. $y$ is the actual label. It comes from the definition of softmax and cross entropy. If you fully solve the derivative of -log(y_hat) w.r.t. theta it equals (y_hat - y). That part is not really relevant (that was essentially given as part of a previous derivation). I only included it here because it does lead into the transpose of W_2. The circle symbol denotes element-wise multiplication. $\endgroup$ – B_Miner Apr 9 '16 at 1:19
  • $\begingroup$ Thank you for reformatting - my latex skills are poor. It took me an hour and I couldnt get the partials right :) $\endgroup$ – B_Miner Apr 9 '16 at 1:23

Allow me to restate the problem in terms of column vectors instead of row vectors $$\eqalign{ a &= W_1^Tx + b_1 &\implies da = W_1^Tdx \cr h &= \sigma(a) &\implies dh = (H-H^2)\,da,\,\,\,\,&H={\rm Diag}(h) \cr \theta &= W_2^Th + b_2 &\implies d\theta = W_2^Tdh \cr y &= {\rm softmax}(\theta) &\implies dy = (Y-yy^T)\,d\theta,\,\,\,\,&Y={\rm Diag}(y) \cr L &= -p:\log y &\implies (p,y) \doteq (y,{\hat y}) \cr }$$ Find the differential of the final (cross entropy) term, and then its gradient $$\eqalign{ dL &= -p:Y^{-1}dy \cr &= -p:Y^{-1}(Y-yy^T)d\theta \cr &= -p:(I-1y^T)d\theta \cr &= (y1^T-I)p:d\theta \cr &= (y-p):W_2^Tdh \cr &= W_2(y-p):(H-H^2)da \cr &= (H-H^2)W_2(y-p):W_1^Tdx \cr &= W_1(H-H^2)W_2(y-p):dx \cr \frac{\partial L}{\partial x} &= W_1(H-H^2)W_2(y-p) \cr }$$ In some of these steps, I used a colon to denote the trace/Frobenius product $$A:B = {\rm tr}(A^TB)$$ Casting the final result back into your preferred notation of row vectors and hats and Hadamard products, yields $$\eqalign{ \frac{\partial L}{\partial x} &= \Big(({\hat y}-y)W_2^T\Big)\circ\Big((h-h\circ h)W_1^T\Big) \cr }$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.