7
$\begingroup$

I saw the following result: $$ \dfrac{\mathrm{d}}{\mathrm{d}x} \left( \log\left( \dfrac{1}{1+\mathrm{e}^{-x}} \right) \right) = \dfrac{1}{\mathrm{e}^x+1} $$ What are the intermediary steps for obtaining this result?

$\endgroup$
3
  • $\begingroup$ Where are you stuck? $\endgroup$ Jun 13 '17 at 11:44
  • $\begingroup$ So I first differentiate the logarithm, according to the rule $log(x)' = 1/x$ and then I differentiate the rest. From the first step I get a $1 + e^{-x}$ which I can not then simplify. $\endgroup$
    – octavian
    Jun 13 '17 at 11:55
  • 1
    $\begingroup$ many software engineers are not super familiar with the math concepts or have forgotten some concepts, and now we see machine learning everywhere! I would like moderators not to just downvoting questions like this. Genuinely many software developers don't have a background in machine learning math and applying ML as a tool. So would appreciate a bit helpful approach! $\endgroup$
    – Exploring
    Oct 21 '20 at 1:11
7
$\begingroup$

Hint:

First, notice that $$ \begin{align} \dfrac{1}{1+e^{-x}} = \dfrac{\mathrm{e}^{x} \cdot 1}{\mathrm{e}^{x} \cdot 1 + \mathrm{e}^{x} \cdot e^{-x}} = \dfrac{\mathrm{e}^{x}}{\mathrm{e}^{x} + 1} \;. \end{align} $$

Second, notice that $$ \begin{align} \ln\left( \dfrac{\mathrm{e}^{x}}{\mathrm{e}^{x} + 1} \right) = \ln\left( \mathrm{e}^{x}\right) - \ln\left( \mathrm{e}^{x} + 1 \right) = x - \ln\left( \mathrm{e}^{x} + 1 \right) \;. \end{align} $$

So, we have $$ \begin{align} \dfrac{\mathrm{d}}{\mathrm{d}x} \ln\left( \dfrac{1}{1+\mathrm{e}^{-x}} \right) &= \dfrac{\mathrm{d}}{\mathrm{d}x} \left( x - \ln\left( \mathrm{e}^{x} + 1 \right) \right) = \dfrac{\mathrm{d}x}{\mathrm{d}x} - \dfrac{\mathrm{d}\ln\left( \mathrm{e}^{x} + 1 \right)}{\mathrm{d}x} \;. \end{align} $$

Can you go on from here using the chain rule?

$\endgroup$
1
  • $\begingroup$ Can you please complete the rest of this explanation? $\endgroup$ Apr 13 at 19:26
5
$\begingroup$

You just have to use the Chain Rule.

$\alpha = 1+e^{-x}$

$\beta = \alpha^{-1}$

$\frac{d\,log(\beta)}{d\,x} = \frac{d\,log(\beta)}{d\,\beta}\,\frac{d\,\beta}{d\,x} = \frac{d\,log(\beta)}{d\,\beta}\,\frac{d\,\alpha^{-1}}{d\,\alpha}\,\frac{d\,\alpha}{d\,x} = \left(\frac{1}{\beta}\right)\,\left(-\frac{1}{\alpha^2}\right)\,\left(-e^{-x}\right) = \frac{e^{-x}}{1+e^{-x}} = \boxed{\frac{1}{e^x + 1}}$

You don't have to worry with signs, because everything in there is always strictly positive.

$\endgroup$
0
$\begingroup$

Log base could refer different bases for different fields. Notice that log(x) refers to base-2 log for computer science, base-e log for mathematical analysis and base-10 log for logarithm tables.

In most general form, derivative of y = logb(1/(1 + ex)) is in following form:

dy/dx = 1 / (ln(b) . (1 + ex))

Of course, if main function were refered to natural logarithm, then b would equal to e, and derivative would be:

dy/dx = 1 / (ln(e) . (1 + ex))

ln(e) would be 1 based on the logarithm of the base rule.

dy/dx = 1 / ((1 + ex))

Mostly, natural logarithm of sigmoid function is mentioned in neural networks. Activation function is calculated in feedforward step whereas its derivative is calculated in backprogation. And derivative of natural log of sigmoid is easier to calculate than other bases.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.