I'm working through Andrej Karpathy's awesome introduction to neural networks and the backpropagation algorithm, and am trying to differentiate the sigmoid function:

$$\sigma(x) = \frac{1}{1+e^{-x}}$$

My understanding is the quotient rule holds that given an equation $y=\frac{t}{b}$ the derivative of the function should be:

$$y' = \frac{t'b - tb'}{b^{2}}$$

After applying the quotient rule to the sigmoid function, I thought the unsimplified result would be:

$$\frac{-e^{-x}}{(1 + e^{-x})^{2}}$$

Because $t'$ is 0, which I thought would yield $0 - tb'$ in the numerator, or $-e^{-x}$. However, Karpathy's unsimplified derivative looks like:

$$\frac{e^{-x}}{(1 + e^{-x})^{2}}$$

Does anyone know why the negative sign in the numerator gets dropped? I'd be very grateful for any help others can offer with this question!

• the derivative of $e^{-x}$ is $-e^{-x}$, together with the minus sign from your formula, they cancel out – barmanthewise Dec 24 '17 at 18:08
• @barmanthewise I thought the derivative of $e^{-x}$ is just $e^{-x}$, but is $e^{-x}$ really $-e^{-x}$? – duhaime Dec 24 '17 at 18:13
• no, thats the derivative if $e^x$, look up a thing called 'chain rule' – barmanthewise Dec 24 '17 at 18:14
• Ah yes, you're quite right, I was using the exponent rule to handle $e^{-x}$ and my book didn't show negative exponent examples. This is exactly what I was overlooking. If you make this an answer I'll mark it right. – duhaime Dec 24 '17 at 18:16
• – Hans Lundmark Dec 24 '17 at 20:38

$$\sigma(x) = \frac{1}{1 + e^{-x}}$$
Use the quotient rule, don't forget about the MINUS sign from the rule, and the MINUS sign due to the derivative of $e^{-x}$.
$$\sigma'(x) = \frac{-(-e^{-x})}{(1 + e^{-x})^2} = \frac{e^{-x}}{(1+e^{-x})^2}$$
better you write $$\sigma(x)=\frac{e^x}{e^x+1}$$ and then the first derivative is given by $$\sigma'(x)=\frac{e^x\cdot (e^x+1)-e^x\cdot e^x}{(e^x+1)^2}$$
• @Dr.SonhardtGraubner I think that's clever. This is exactly the rule I wanted to apply, and multiplying the numerator and den by $e^{x}$ does simplify things a bit. – duhaime Dec 24 '17 at 18:20