I have seen an example (it is in terms of neural network back propagation) that I don’t understand.


  • $\mathbf{a} = \mathbf{x}\mathbf{W}_{1}+\mathbf{b}_{1}$ (where $\mathbf{x}$ is dimension ($1\times5$), $\mathbf{W}_1$ is ($5\times3$)) and $\mathbf{b}_1$ is ($1\times3$))
  • $\textbf{h}=\sigma(\textbf{a})$ is the sigmoid function: $\frac{1}{1+\exp(-a_{i})}$ which acts on the $n$-dimensional vector $a$ element-wise, meaning $\sigma(\mathbf{a}) =[\sigma(a_{1}),\sigma(a_{2}),\ldots,\sigma(a_{n})]$
  • $\theta = \mathbf{h}\mathbf{W}_{2}+\mathbf{b}_2$ (where $\mathbf{h}$ is dimension ($1\times3$), $\mathbf{W}_2$ is ($3\times5$) and $\mathbf{b}_2$ is ($1\times5$))
  • $\hat{\mathbf{y}}= \operatorname{softmax}(\theta)$ (where $\hat{\mathbf{y}}$ is dimension ($1\times5$)) (definition)
  • $L=\operatorname{xent}(y, \hat{y})$ (definition)

The derivative of interest is $\frac{\partial L}{\partial x}$ or by the chain rule:

$$\frac{\partial L}{\partial{x}} =\frac{\partial L}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial{\theta}}\frac{\partial{\theta}}{\partial {h}}\frac{\partial{h}}{\partial{a}}\frac{\partial{a}}{\partial{x}}$$

The result they show makes perfect sense to me (almost)

$((\hat{\mathbf{y}}-\mathbf{y}) \mathbf{W}_{2}^{T})\circ\sigma'(a)\mathbf{W}_{1}$

My Questions:

  1. Since $(\hat{\mathbf{y}}-\mathbf{y})$ is dimension ($1\times5$) they transpose $\mathbf{W}_{2}$ to conform to vector matrix multiplication. Is this OK? Can you just transpose a matrix when you want?
  2. Why the elementwise multiplication by the derivative of $\sigma(a)$ The rationale is that since $\sigma$ is an elementwise operator, this is proper. I don’t understand why you would not apply sigma to each element of $\mathbf{a}$ and then matrix-multiply this result against the vector on the left?
  • $\begingroup$ I tried to reformat a little the equations. The solution has a ${\bf y}$ which does not appear in the formulas before. What's that? Further, the $\log$ is to be taken elemenwise? Further, what's $\circ$ ? $\endgroup$
    – leonbloy
    Apr 8, 2016 at 23:26
  • $\begingroup$ Hi. $y$ is the actual label. It comes from the definition of softmax and cross entropy. If you fully solve the derivative of -log(y_hat) w.r.t. theta it equals (y_hat - y). That part is not really relevant (that was essentially given as part of a previous derivation). I only included it here because it does lead into the transpose of W_2. The circle symbol denotes element-wise multiplication. $\endgroup$
    – B_Miner
    Apr 9, 2016 at 1:19
  • $\begingroup$ Thank you for reformatting - my latex skills are poor. It took me an hour and I couldnt get the partials right :) $\endgroup$
    – B_Miner
    Apr 9, 2016 at 1:23

1 Answer 1


Allow me to restate the problem in terms of column vectors instead of row vectors $$\eqalign{ a &= W_1^Tx + b_1 &\implies da = W_1^Tdx \cr h &= \sigma(a) &\implies dh = (H-H^2)\,da,\,\,\,\,&H={\rm Diag}(h) \cr \theta &= W_2^Th + b_2 &\implies d\theta = W_2^Tdh \cr y &= {\rm softmax}(\theta) &\implies dy = (Y-yy^T)\,d\theta,\,\,\,\,&Y={\rm Diag}(y) \cr L &= -p:\log y &\implies (p,y) \doteq (y,{\hat y}) \cr }$$ Find the differential of the final (cross entropy) term, and then its gradient $$\eqalign{ dL &= -p:Y^{-1}dy \cr &= -p:Y^{-1}(Y-yy^T)d\theta \cr &= -p:(I-1y^T)d\theta \cr &= (y1^T-I)p:d\theta \cr &= (y-p):W_2^Tdh \cr &= W_2(y-p):(H-H^2)da \cr &= (H-H^2)W_2(y-p):W_1^Tdx \cr &= W_1(H-H^2)W_2(y-p):dx \cr \frac{\partial L}{\partial x} &= W_1(H-H^2)W_2(y-p) \cr }$$ In some of these steps, I used a colon to denote the trace/Frobenius product $$A:B = {\rm tr}(A^TB)$$ Casting the final result back into your preferred notation of row vectors and hats and Hadamard products, yields $$\eqalign{ \frac{\partial L}{\partial x} &= \Big(({\hat y}-y)W_2^T\Big)\circ\Big((h-h\circ h)W_1^T\Big) \cr }$$


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .