132
$\begingroup$

In my AI textbook there is this paragraph, without any explanation.

The sigmoid function is defined as follows

$$\sigma (x) = \frac{1}{1+e^{-x}}.$$

This function is easy to differentiate because

$$\frac{d\sigma (x)}{d(x)} = \sigma (x)\cdot (1-\sigma(x)).$$

It has been a long time since I've taken differential equations, so could anyone tell me how they got from the first equation to the second?

$\endgroup$
  • 2
    $\begingroup$ What AI textbook is that? $\endgroup$ – frog1944 Sep 29 '17 at 11:28
  • 3
    $\begingroup$ @frog1944: It seems to be Artificial Intelligence Illuminated by Ben Coppin, page 302 (Google Books link). $\endgroup$ – Hans Lundmark Nov 6 '17 at 11:18
  • 1
    $\begingroup$ @HansLundmark thank you very much! $\endgroup$ – frog1944 Nov 6 '17 at 19:06
  • 1
    $\begingroup$ Any book on neural networks will deal with the sigmoid function. It is useful because of the simple way backpropagation works; a lot of computing work is saved when training a network from a set of results. In nature, other functions are possible, like arctan, rational functions, and more. $\endgroup$ – richard1941 Jan 18 '18 at 2:02
  • $\begingroup$ One of the reasons they use the sigmoid is that it is easy to differentiate and facilitates backpropagation. Not so for other candidates like sign(x), arctangent(x), sinh(x), etc. $\endgroup$ – richard1941 Dec 21 '18 at 18:22
93
$\begingroup$

Consider $$ f(x)=\dfrac{1}{\sigma(x)} = 1+e^{-x} . $$ Then, on the one hand, the chain rule gives $$ f'(x) = \frac{d}{dx} \biggl( \frac{1}{\sigma(x)} \biggr) = -\frac{\sigma'(x)}{\sigma(x)^2} , $$ and on the other hand, $$ f'(x) = \frac{d}{dx} \bigl( 1+e^{-x} \bigr) = -e^{-x} = 1-f(x) = 1 - \frac{1}{\sigma(x)} = \frac{\sigma(x)-1}{\sigma(x)} . $$ Equate the two expressions, and voilà!

(Cf. also this answer to a very recent question.)

$\endgroup$
  • $\begingroup$ How do you derive 1 + e^-x as -e^-x? (Update: I think it's because the derivative of e^x = e^x) en.wikipedia.org/wiki/Derivative#Rules_for_basic_functions $\endgroup$ – Adam Grant Aug 4 '17 at 23:36
  • 1
    $\begingroup$ @AdamGrant: Yes, since then the chain rule gives $e^{kx}=k e^{kx}$ for any constant $k$. (In this case, we have $k=-1$.) $\endgroup$ – Hans Lundmark Aug 5 '17 at 6:51
  • $\begingroup$ Correction to silly typo in the previous comment: it should be $\frac{d}{dx} e^{kx} = k e^{kx}$, of course. $\endgroup$ – Hans Lundmark Jan 24 at 7:18
231
$\begingroup$

Let's denote the sigmoid function as $\sigma(x) = \dfrac{1}{1 + e^{-x}}$.

The derivative of the sigmoid is $\dfrac{d}{dx}\sigma(x) = \sigma(x)(1 - \sigma(x))$.

Here's a detailed derivation:

$$ \begin{align} \dfrac{d}{dx} \sigma(x) &= \dfrac{d}{dx} \left[ \dfrac{1}{1 + e^{-x}} \right] \\ &= \dfrac{d}{dx} \left( 1 + \mathrm{e}^{-x} \right)^{-1} \\ &= -(1 + e^{-x})^{-2}(-e^{-x}) \\ &= \dfrac{e^{-x}}{\left(1 + e^{-x}\right)^2} \\ &= \dfrac{1}{1 + e^{-x}\ } \cdot \dfrac{e^{-x}}{1 + e^{-x}} \\ &= \dfrac{1}{1 + e^{-x}\ } \cdot \dfrac{(1 + e^{-x}) - 1}{1 + e^{-x}} \\ &= \dfrac{1}{1 + e^{-x}\ } \cdot \left( \dfrac{1 + e^{-x}}{1 + e^{-x}} - \dfrac{1}{1 + e^{-x}} \right) \\ &= \dfrac{1}{1 + e^{-x}\ } \cdot \left( 1 - \dfrac{1}{1 + e^{-x}} \right) \\ &= \sigma(x) \cdot (1 - \sigma(x)) \end{align} $$

$\endgroup$
  • 3
    $\begingroup$ Sir, Is d(e^x)=e^x? $\endgroup$ – Ravinder Payal Mar 6 '17 at 11:47
  • 3
    $\begingroup$ @RavinderPayal: Yes, d/dx(e^x) = e^x; If you want a proof of that see: khanacademy.org/math/ap-calculus-ab/advanced-differentiation-ab/… $\endgroup$ – Mike Percy Mar 6 '17 at 23:02
  • 1
    $\begingroup$ Where does the (1 + e^-x) - 1 suddenly come from in third row up from bottom? $\endgroup$ – Jarad Jul 11 '17 at 5:26
  • 3
    $\begingroup$ @Jarad: e^-x == 1 + e^-x - 1; we are just adding 1 and subtracting 1 from the same term, which changes nothing. $\endgroup$ – Mike Percy Aug 17 '17 at 9:27
  • 11
    $\begingroup$ Thank you. Your explanation is much better than the answer. $\endgroup$ – Dmitriy Dokshin Sep 21 '17 at 19:03
16
$\begingroup$

Note that from your given equation,

$(1+e^{-x})\sigma=1$

$\Rightarrow -e^{-x}\sigma+(1+e^{-x})\frac{d\sigma}{dx}=0$ (differentiating using product rule)

$\Rightarrow \frac{d\sigma}{dx}=\sigma.\frac{e^{-x}}{(1+e^{-x})}=\sigma.\frac{(1+e^{-x})-1}{(1+e^{-x})}=\sigma.\left[1-\frac{1}{(1+e^{-x})}\right]=\sigma.(1-\sigma)$

$\endgroup$
6
$\begingroup$

Since $\sigma(x)$ is a composite function, firstly we need to use chain rule to dig down to the x term, then we can factor back to the $\sigma(x)$ fuction: $$ \begin{align} \frac{d}{dx}\sigma(x) &= (\frac{1}{1+e^{-x}})' \\ &= -\frac{1}{(1+e^{-x})^{2}} \cdot (1) \cdot -e^{-x} \\ &= \frac{e^{-x}}{(1+e^{-x})^{2}}, \\ \because \sigma(x) &= \frac{1}{1+e^{-x}}, \\ e^{-x} &= \frac{1 - \sigma(x)}{\sigma(x)}, \\ 1+e^{-x} &= \frac{1}{\sigma(x)}; \\ \therefore \frac{d}{dx}\sigma(x) &= \frac{\frac{1 - \sigma(x)}{\sigma(x)}}{(\frac{1}{\sigma(x)})^{2}} \\ &= (1 - \sigma(x)) \cdot \sigma(x) \end{align}$$

$\endgroup$
5
$\begingroup$

Let's say we want to find the derivative of $y=σ(x)=(1+\exp(−x))^{−1}$. So we have:

$$ \begin{align} \frac{dy}{dx} & = (-1)(1 + \exp(-x))^{-2} \frac{d}{dx}(1 + \exp(-x)) \\ \\ & = (-1)(1 + \exp(-x))^{-2}(0 + \frac{d}{dx}\exp(-x)) \\ \\ & = (-1)(1 + \exp(-x))^{-2}(\exp(-x)) \frac{d}{dx}(-x) \\ \\ & = (-1)(1 + \exp(-x))^{-2}(\exp(-x))(-1) \\ \\ & = \frac{\exp(-x)} {(1 + \exp(-x))^2} \\ \\ & = \frac{1 + \exp(-x) -1} {(1 + \exp(-x))^2} \\ \\ & = \frac{1 + \exp(-x)} {(1 + \exp(-x))^2} - \frac{1} {(1 + \exp(-x))^2} \\ \\ & = \sigma(x) - (\sigma(x))^2 \\ \\ & = \sigma(x) \cdot (1 - \sigma(x)) \end{align} $$

$\endgroup$
2
$\begingroup$

By directly differenting:

$$ \sigma^{'} (x)= \frac{1. e^{-x}}{(1+e^{-x})^2} $$

Separately compute, multiply:

$${\sigma(x)}.{(1-\sigma(x))} =\frac{ e^{-x}}{(1+e^{-x}) } . \frac{ 1}{(1+e^{-x}) }$$

The RHSs agree.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.