247
$\begingroup$

In my AI textbook there is this paragraph, without any explanation.

The sigmoid function is defined as follows

$$\sigma (x) = \frac{1}{1+e^{-x}}.$$

This function is easy to differentiate because

$$\frac{d\sigma (x)}{d(x)} = \sigma (x)\cdot (1-\sigma(x)).$$

It has been a long time since I've taken differential equations, so could anyone tell me how they got from the first equation to the second?

$\endgroup$
5
  • 2
    $\begingroup$ What AI textbook is that? $\endgroup$
    – frog1944
    Sep 29, 2017 at 11:28
  • 5
    $\begingroup$ @frog1944: It seems to be Artificial Intelligence Illuminated by Ben Coppin, page 302 (Google Books link). $\endgroup$ Nov 6, 2017 at 11:18
  • 1
    $\begingroup$ @HansLundmark thank you very much! $\endgroup$
    – frog1944
    Nov 6, 2017 at 19:06
  • 1
    $\begingroup$ Any book on neural networks will deal with the sigmoid function. It is useful because of the simple way backpropagation works; a lot of computing work is saved when training a network from a set of results. In nature, other functions are possible, like arctan, rational functions, and more. $\endgroup$ Jan 18, 2018 at 2:02
  • 1
    $\begingroup$ One of the reasons they use the sigmoid is that it is easy to differentiate and facilitates backpropagation. Not so for other candidates like sign(x), arctangent(x), sinh(x), etc. $\endgroup$ Dec 21, 2018 at 18:22

10 Answers 10

403
$\begingroup$

Let's denote the sigmoid function as $\sigma(x) = \dfrac{1}{1 + e^{-x}}$.

The derivative of the sigmoid is $\dfrac{d}{dx}\sigma(x) = \sigma(x)(1 - \sigma(x))$.

Here's a detailed derivation:

$$ \begin{align} \dfrac{d}{dx} \sigma(x) &= \dfrac{d}{dx} \left[ \dfrac{1}{1 + e^{-x}} \right] \\ &= \dfrac{d}{dx} \left( 1 + \mathrm{e}^{-x} \right)^{-1} \\ &= -(1 + e^{-x})^{-2}(-e^{-x}) \\ &= \dfrac{e^{-x}}{\left(1 + e^{-x}\right)^2} \\ &= \dfrac{1}{1 + e^{-x}\ } \cdot \dfrac{e^{-x}}{1 + e^{-x}} \\ &= \dfrac{1}{1 + e^{-x}\ } \cdot \dfrac{(1 + e^{-x}) - 1}{1 + e^{-x}} \\ &= \dfrac{1}{1 + e^{-x}\ } \cdot \left( \dfrac{1 + e^{-x}}{1 + e^{-x}} - \dfrac{1}{1 + e^{-x}} \right) \\ &= \dfrac{1}{1 + e^{-x}\ } \cdot \left( 1 - \dfrac{1}{1 + e^{-x}} \right) \\ &= \sigma(x) \cdot (1 - \sigma(x)) \end{align} $$

$\endgroup$
11
  • 5
    $\begingroup$ Sir, Is d(e^x)=e^x? $\endgroup$ Mar 6, 2017 at 11:47
  • 6
    $\begingroup$ @RavinderPayal: Yes, d/dx(e^x) = e^x; If you want a proof of that see: khanacademy.org/math/ap-calculus-ab/advanced-differentiation-ab/… $\endgroup$ Mar 6, 2017 at 23:02
  • 1
    $\begingroup$ Where does the (1 + e^-x) - 1 suddenly come from in third row up from bottom? $\endgroup$
    – Jarad
    Jul 11, 2017 at 5:26
  • 5
    $\begingroup$ @Jarad: e^-x == 1 + e^-x - 1; we are just adding 1 and subtracting 1 from the same term, which changes nothing. $\endgroup$ Aug 17, 2017 at 9:27
  • 15
    $\begingroup$ Thank you. Your explanation is much better than the answer. $\endgroup$ Sep 21, 2017 at 19:03
117
$\begingroup$

Consider $$ f(x)=\dfrac{1}{\sigma(x)} = 1+e^{-x} . $$ Then, on the one hand, the chain rule gives $$ f'(x) = \frac{d}{dx} \biggl( \frac{1}{\sigma(x)} \biggr) = -\frac{\sigma'(x)}{\sigma(x)^2} , $$ and on the other hand, $$ f'(x) = \frac{d}{dx} \bigl( 1+e^{-x} \bigr) = -e^{-x} = 1-f(x) = 1 - \frac{1}{\sigma(x)} = \frac{\sigma(x)-1}{\sigma(x)} . $$ Equate the two expressions, and voilà!

(Cf. also this answer.)

$\endgroup$
3
  • $\begingroup$ How do you derive 1 + e^-x as -e^-x? (Update: I think it's because the derivative of e^x = e^x) en.wikipedia.org/wiki/Derivative#Rules_for_basic_functions $\endgroup$
    – Adam Grant
    Aug 4, 2017 at 23:36
  • 2
    $\begingroup$ @AdamGrant: Yes, since then the chain rule gives $e^{kx}=k e^{kx}$ for any constant $k$. (In this case, we have $k=-1$.) $\endgroup$ Aug 5, 2017 at 6:51
  • $\begingroup$ Correction to silly typo in the previous comment: it should be $\frac{d}{dx} e^{kx} = k e^{kx}$, of course. $\endgroup$ Jan 24, 2019 at 7:18
22
$\begingroup$

Note that from your given equation,

$(1+e^{-x})\sigma=1$

$\Rightarrow -e^{-x}\sigma+(1+e^{-x})\frac{d\sigma}{dx}=0$ (differentiating using product rule)

$\Rightarrow \frac{d\sigma}{dx}=\sigma.\frac{e^{-x}}{(1+e^{-x})}=\sigma.\frac{(1+e^{-x})-1}{(1+e^{-x})}=\sigma.\left[1-\frac{1}{(1+e^{-x})}\right]=\sigma.(1-\sigma)$

$\endgroup$
0
8
$\begingroup$

Since $\sigma(x)$ is a composite function, firstly we need to use chain rule to dig down to the x term, then we can factor back to the $\sigma(x)$ fuction: $$ \begin{align} \frac{d}{dx}\sigma(x) &= (\frac{1}{1+e^{-x}})' \\ &= -\frac{1}{(1+e^{-x})^{2}} \cdot (1) \cdot -e^{-x} \\ &= \frac{e^{-x}}{(1+e^{-x})^{2}}, \\ \because \sigma(x) &= \frac{1}{1+e^{-x}}, \\ e^{-x} &= \frac{1 - \sigma(x)}{\sigma(x)}, \\ 1+e^{-x} &= \frac{1}{\sigma(x)}; \\ \therefore \frac{d}{dx}\sigma(x) &= \frac{\frac{1 - \sigma(x)}{\sigma(x)}}{(\frac{1}{\sigma(x)})^{2}} \\ &= (1 - \sigma(x)) \cdot \sigma(x) \end{align}$$

$\endgroup$
6
$\begingroup$

Let's say we want to find the derivative of $y=σ(x)=(1+\exp(−x))^{−1}$. So we have:

$$ \begin{align} \frac{dy}{dx} & = (-1)(1 + \exp(-x))^{-2} \frac{d}{dx}(1 + \exp(-x)) \\ \\ & = (-1)(1 + \exp(-x))^{-2}(0 + \frac{d}{dx}\exp(-x)) \\ \\ & = (-1)(1 + \exp(-x))^{-2}(\exp(-x)) \frac{d}{dx}(-x) \\ \\ & = (-1)(1 + \exp(-x))^{-2}(\exp(-x))(-1) \\ \\ & = \frac{\exp(-x)} {(1 + \exp(-x))^2} \\ \\ & = \frac{1 + \exp(-x) -1} {(1 + \exp(-x))^2} \\ \\ & = \frac{1 + \exp(-x)} {(1 + \exp(-x))^2} - \frac{1} {(1 + \exp(-x))^2} \\ \\ & = \sigma(x) - (\sigma(x))^2 \\ \\ & = \sigma(x) \cdot (1 - \sigma(x)) \end{align} $$

$\endgroup$
5
$\begingroup$

By directly differenting:

$$ \sigma^{'} (x)= \frac{1. e^{-x}}{(1+e^{-x})^2} $$

Separately compute, multiply:

$${\sigma(x)}.{(1-\sigma(x))} =\frac{ e^{-x}}{(1+e^{-x}) } . \frac{ 1}{(1+e^{-x}) }$$

The RHSs agree.

EDIT1:

In general a solution of differential equation

$$ \frac{dy}{dx}=y(1-y) $$

can be seen to be

$$\frac{1}{1+c e^{-x}} \rightarrow \frac{1}{1+ e^{-x}} $$

with center point integration constant evaluated at $x=0, y=\frac12;\, c=1. $

$\endgroup$
0
$\begingroup$

No answer yet involves the logarithm. If $\log$ denotes the natural logarithm then by the chain rule $$\frac{d}{dx} \log(\sigma(x)) = \frac{1}{\sigma(x)}\frac{d \sigma(x)}{dx}. $$ Furthermore $\log(\sigma(x))=\log(e^x) -\log(1+e^x)=x - \log(1+e^x)$ so that $$\frac{d}{dx} \log(\sigma(x)) = 1 - \frac{e^x}{1+e^x} = 1-\sigma(x).$$ Equaling the two displays gives $(d/dx)\sigma(x) = \sigma(x)(1-\sigma(x))$ as desired.

$\endgroup$
0
$\begingroup$

Another approach using the quotient rule is as follows:

let $\sigma=\frac{1}{1+e^-x}$ upon rearranging ,

$\sigma=\frac{e^x}{1+e^x}$

using the quotient rule

$\frac{d}{{dx}}\left( {\frac{{f\left( x \right)}}{{g\left( x \right)}}} \right) = \frac{{\frac{d}{{dx}}f\left( x \right)g\left( x \right) - f\left( x \right)\frac{d}{{dx}}g\left( x \right)}}{{g^2 \left( x \right)}}$,

let $f(x)=e^x$ and $g(x)=1+e^x$,

we get $\frac{d\sigma}{dx}=\frac{(e^x)(1+e^x)-e^xe^x}{(1+e^x)^2}$

upon rearraning, we get :

$\frac{d\sigma}{dx}=\frac{e^x}{1+e^x}\frac{(1+e^x)-e^x}{1+e^x}$

upon further rearrangement:

$\frac{d\sigma}{dx}=\frac{e^x}{1+e^x}(1-\frac{e^x}{1+e^x})$

$\frac{d\sigma}{dx}=\sigma(1-\sigma)$

$\endgroup$
1
  • $\begingroup$ Good work ${}{}{}{}$ $\endgroup$
    – Some Guy
    Feb 26, 2021 at 6:29
-1
$\begingroup$

$\exp(-x) = \frac{1}{\sigma} -1 $ (By definition). Take the derivative of both sides:

$-\exp(-x)= -\frac{\sigma'}{\sigma^2}$

Add the two to get: $0 = \frac{1}{\sigma} -1 -\frac{\sigma'}{\sigma^2}$

and solve for $\sigma'=\sigma(1-\sigma)$ qed

$\endgroup$
-3
$\begingroup$

Using my HP Prime, I differentiated 1/(1+exp(-x)) to get exp(-x)/(1+exp(-x))^2. Factor out 1/(1+exp(-x), which is sigma(x), and the rest is 1-sigma(x). That is proof by calculator. Beware! That machine can become addictive because of the way it amplifies your capability.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.