I'm working through Andrej Karpathy's awesome introduction to neural networks and the backpropagation algorithm, and am trying to differentiate the sigmoid function:

$$ \sigma(x) = \frac{1}{1+e^{-x}} $$

My understanding is the quotient rule holds that given an equation $y=\frac{t}{b}$ the derivative of the function should be:

$$y' = \frac{t'b - tb'}{b^{2}}$$

After applying the quotient rule to the sigmoid function, I thought the unsimplified result would be:

$$ \frac{-e^{-x}}{(1 + e^{-x})^{2}} $$

Because $t'$ is 0, which I thought would yield $ 0 - tb' $ in the numerator, or $-e^{-x}$. However, Karpathy's unsimplified derivative looks like:

$$ \frac{e^{-x}}{(1 + e^{-x})^{2}} $$

Does anyone know why the negative sign in the numerator gets dropped? I'd be very grateful for any help others can offer with this question!

  • $\begingroup$ the derivative of $e^{-x}$ is $-e^{-x}$, together with the minus sign from your formula, they cancel out $\endgroup$ – barmanthewise Dec 24 '17 at 18:08
  • $\begingroup$ @barmanthewise I thought the derivative of $e^{-x}$ is just $e^{-x}$, but is $e^{-x}$ really $-e^{-x}$? $\endgroup$ – duhaime Dec 24 '17 at 18:13
  • $\begingroup$ no, thats the derivative if $e^x$, look up a thing called 'chain rule' $\endgroup$ – barmanthewise Dec 24 '17 at 18:14
  • $\begingroup$ Ah yes, you're quite right, I was using the exponent rule to handle $e^{-x}$ and my book didn't show negative exponent examples. This is exactly what I was overlooking. If you make this an answer I'll mark it right. $\endgroup$ – duhaime Dec 24 '17 at 18:16
  • $\begingroup$ Related: math.stackexchange.com/questions/78575/… $\endgroup$ – Hans Lundmark Dec 24 '17 at 20:38

$$\sigma(x) = \frac{1}{1 + e^{-x}}$$

Use the quotient rule, don't forget about the MINUS sign from the rule, and the MINUS sign due to the derivative of $e^{-x}$.

$$\sigma'(x) = \frac{-(-e^{-x})}{(1 + e^{-x})^2} = \frac{e^{-x}}{(1+e^{-x})^2}$$

  • 1
    $\begingroup$ I had missed the second minus sign, thank you for spelling it out. $\endgroup$ – duhaime Dec 24 '17 at 18:21

better you write $$\sigma(x)=\frac{e^x}{e^x+1}$$ and then the first derivative is given by $$\sigma'(x)=\frac{e^x\cdot (e^x+1)-e^x\cdot e^x}{(e^x+1)^2}$$

  • $\begingroup$ @Dr.SonhardtGraubner I think that's clever. This is exactly the rule I wanted to apply, and multiplying the numerator and den by $e^{x}$ does simplify things a bit. $\endgroup$ – duhaime Dec 24 '17 at 18:20
  • 1
    $\begingroup$ and you can not lose a minus sign $\endgroup$ – Dr. Sonnhard Graubner Dec 24 '17 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.