0
$\begingroup$

Definition of logistic sigmoid function is:

$$σ(t)=\frac{1}{1 + e^{-t}}$$

Differentiating this is very simple, It just requires simplification with exponent rule and differentiation with quotient rule.

$$\frac{d}{dx}\frac{1}{1 + e^{-t}}=\frac{d}{dx}\frac{e^{t}}{1 + e^{t}}=\frac{e^{t}(1 + e^{t}) -e^{2t}}{(1 + e^{t})^{2}}=\frac{e^{t}}{(1 + e^{t})^{2}}$$


But what if instead of $t$, the input was weighted sum $\sum{w_ix_i}$ ?

The partial derivative with respect to $w_i$ somehow becomes $σ*1-σ*x_{i}$, this is how one person calculated it:

$$\frac{\partial}{w_{i}}\frac{1}{1 + e^{-\sum{w_ix_i}}}=\frac{-1}{(1+e^{-\sum{w_ix_i}})^{2}}*e^{-\sum{w_ix_i}}*(-x_i)$$ $$\frac{\partial}{w_{i}}\frac{1}{1 + e^{-\sum{w_ix_i}}}=\frac{1}{(1+e^{-\sum{w_ix_i}})}*\frac{e^{-\sum{w_ix_i}}}{{(1+e^{-\sum{w_ix_i}})}}*x_i$$

Now, considering that $1-σ$ is $\frac{e^{-\sum{w_ix_i}}}{{(1+e^{-\sum{w_ix_i}})}}$, we get:

$$\frac{\partial}{w_{i}}\frac{1}{1 + e^{-\sum{w_ix_i}}}=σ*1-σ*x_i$$

Is this derivative properly calculated? If so how exactly? $x_i$ is somehow outside of summation, is it due to it being treated as constant?

$\endgroup$
1
$\begingroup$

You have a composite function, $\sigma=1/(1-e^{-t})$ where $t=\sum w_i x_i$, so just use the chain rule: $$ \frac{\partial \sigma}{\partial w_i} = \frac{d \sigma}{d t} \cdot \frac{\partial t}{\partial w_i} = \sigma (1-\sigma) \cdot x_i . $$ (See here for a slick way of deriving $\frac{d \sigma}{d t}=\sigma (1-\sigma)$.)

$\endgroup$
  • $\begingroup$ Thank you for the answer. I assume that $x$ is out due to linearity property due to being treated as constant correct? $\endgroup$ – ShellRox Aug 8 '18 at 20:34
  • 1
    $\begingroup$ Yes, I'm assuming that the $x_i$ are constants. $\endgroup$ – Hans Lundmark Aug 9 '18 at 10:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.