I have been working on programming a feed forward neural network that uses stochastic gradient descent and I am still a little confused on all the calculus. To make sure I have the math correct, I am using the following network as an example: Neural network image

The equation for the output of the network should be the following I believe ( s(x) is sigmoid ):

$$ s\left(x\right)\ =\ \frac{1}{1+e^{-x}} $$ $$ O=s\left(s\left(s\left(Iw_1+1\right)w_2+1\right)w_4+s\left(s\left(Iw_1+1\right)w_3+1\right)w_5+1\right) $$

When I did the math to get the derivative of O with respect to w1, I got: ( d(x) is the derivative of sigmoid ) $$ d\left(x\right)=s\left(x\right)\left(1-s\left(x\right)\right) $$

$$ d\left(s\left(s\left(Iw_1+1\right)w_2+1\right)w_4+s\left(s\left(Iw_1+1\right)w_3+1\right)w_5+1\right)\cdot\left(\left(d\left(s\left(Iw_1+1\right)w_2+1\right)\cdot d\left(Iw_1+1\right)\cdot I\right)+\left(d\left(s\left(Iw_1+1\right)w_3+1\right)\cdot d\left(Iw_1+1\right)\cdot I\right)\right) $$

I used https://www.desmos.com/calculator to see if it got the same answer for the derivative of O with respect to w1. With all the weights set to 0.5, all the biases set to 1, and the input set to 1, desmos said that the derivative was 0.0014291881022. But my equation gave 0.00571675240882. Is there a mistake somewhere in my math or is some weird thing desmos does? Sorry if I did anything simple wrong or messed up the notation. I am still very new to calculus.


Looks like you're missing some components in the second term. The full expression should be $$ \frac{\partial O}{\partial w_1} =d(s(s(Iw_1+1)w_2+1)w_4+s(s(Iw_1+1)w_3+1)w_5+1)⋅[(\mathbf{w_4}d(s(Iw_1+1)w_2+1)⋅\mathbf{w_2}d(Iw_1+1)⋅I)+(\mathbf{w_5}d(s(Iw_1+1)w_3+1)⋅\mathbf{w_3}d(Iw_1+1)⋅I)], $$ with the missing terms in bold.

Since you set all the weights to 0.5 in your check, you'll see that the missing terms $w_4w_2$ and $w_5w_3$ will both multiply to 0.25. And indeed, your answer is off by a factor of 0.25


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.