2
$\begingroup$

I am attempting to calculate the partial derivative of the sigmoid function with respect to theta:

$ y = \frac{1}{1+ e^{-\theta x}}$

Let:

$v = -\theta x $

$u = (1 + e^{-\theta x}) = (1 + e^v)$

Then:

$ \frac{\partial y}{\partial u} = -u^{-2}$

$ \frac{\partial u}{\partial v} = e^v $

$ \frac{\partial v}{\partial \theta_i} = -x_i $

So, applying the chain rule:

$ \frac{\partial y}{\partial \theta_i} $

$= \frac{\partial y}{\partial u} \frac{\partial u}{\partial v} \frac{\partial v}{\partial \theta_i}$

$= -u^{-2} e^v (-x_i)$

$= -(1 + e^v)^{-2} e^v (-x_i)$

$= -(1+e^{-\theta x})^{-2} e^{-\theta x} (-x_i)$

$=\frac{-x_ie^{-\theta x}}{-(1+e^{-\theta x})^2} $

At this point, I'm trying to figure out how to get it into this form:

$ \frac{\partial y}{\partial \theta_i} = y(1 - y)$

How do I accomplish this?

Also, it is alleged that:

$1 - \frac{1}{1+ e^{-\theta x}} = \frac{e^{-\theta x}}{1 + e^{-\theta x}}$

How is this possible?

$\endgroup$
1
  • $\begingroup$ Are you sure that's actually possible. Best I can see, that derivative should be coming out to $xy(1-y)$. $\endgroup$ Commented Apr 5, 2017 at 21:52

1 Answer 1

2
$\begingroup$

Let $$ f(\theta)=\frac{1}{g(\theta)}=\frac{1}{1+e^{h(\theta)}}=\frac{1}{1+e^{-\theta x}}. $$ First, note that by the Chain rule \begin{align*} f^{\prime}(\theta) & =-g^{\prime}(\theta)/(g(\theta))^{2},\\ g^{\prime}(\theta) & =e^{h(\theta)}h^{\prime}(\theta),\\ \text{and }h^{\prime}(\theta) & =-x. \end{align*} Putting this all together, $$ f^{\prime}(\theta)=-\frac{g^{\prime}(\theta)}{(g(\theta))^{2}}=-\frac{e^{h(\theta)}h^{\prime}(\theta)}{(g(\theta))^{2}}=\frac{e^{-\theta x}x}{(1+e^{-\theta x})^{2}}. $$ Moreover, note that $$ 1-f(\theta)=1-\frac{1}{1+e^{-\theta x}}=\frac{1+e^{-\theta x}}{1+e^{-\theta x}}-\frac{1}{1+e^{-\theta x}}=\frac{1+e^{-\theta x}-1}{1+e^{-\theta x}}=\frac{e^{-\theta x}}{1+e^{-\theta x}}. $$ Therefore, $$ f^{\prime}(\theta)=\frac{1}{1+e^{-\theta x}}\frac{e^{-\theta x}}{1+e^{-\theta x}}x=f(\theta)\left(1-f(\theta)\right)x. $$

$\endgroup$
2
  • $\begingroup$ In your last line, you still have an $x$ in the numerator that isn't present in the value of $1-f(\theta)$. Am I missing something? $\endgroup$ Commented Apr 5, 2017 at 21:50
  • $\begingroup$ Good catch; that was a mistake on my part. I guess the $x$ is there to stay. Perhaps the question wanted OP to write it in the form $f(\theta)(1-f(\theta))$ for $x=1$? (since, otherwise, it is simply not true) $\endgroup$
    – parsiad
    Commented Apr 5, 2017 at 21:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .