0
$\begingroup$

Let's consider the product $\lvert w \sigma^{'}(z) \rvert=\lvert w \sigma^{'}(wa+b) \rvert$ where $\sigma(z)={1 \over 1+ e^{-z}}$ is the sigmoid function. We know $\sigma^{'}(wa+b) \leq {1 \over 4}$ and so we can only have $\lvert w \sigma^{'}(wa+b) \rvert \geq 1$ when $\lvert w \rvert \geq 4$. Given this, we want to show that the interval of $a$ satisfying $\lvert w \sigma^{'}(wa+b) \rvert \geq 1$ is no greater in width than $${2 \over \lvert w \rvert} \ln{\left( {{\lvert w \rvert\left(1+\sqrt{1-{4 \over \lvert w \rvert}}\right)} \over 2}-1\right)}$$ To show this, I use $\sigma^{'}(z)={e^{-z} \over (1+e^{-z})^2}>0$ to get $${e^{-z} \over (1+e^{-z})^2} \geq {1 \over \lvert w \rvert}$$ $$e^{-2z}+(2 -\lvert w \rvert)e^{-z}+1 \leq 0$$ $\Delta = \lvert w \rvert (\lvert w \rvert - 4) \geq 0$, so the solutions to the inequality are between $$e^{-z} = {{\lvert w \rvert\left(1\pm\sqrt{1-{4 \over \lvert w \rvert}}\right)} \over 2}-1$$ and we have $a$ between $$-{1 \over w}\ln{\left( {{\lvert w \rvert\left(1\pm\sqrt{1-{4 \over \lvert w \rvert}}\right)} \over 2}-1\right)}-{b \over w}$$ I don't quite see how I can relate this to the expected result and would appreciate any hints.

$\endgroup$
0
$\begingroup$

Let $$x_{1,2}={{\lvert w \rvert\left(1\pm\sqrt{1-{4 \over \lvert w \rvert}}\right)} \over 2}-1$$ be the roots of the equation $$x^2+(2 -\lvert w \rvert)x+1=0.$$ By Vieta’s formula $x_2=1/x_1$. So

$$\frac {-1}w(\ln x_1+b-\ln x_2-b)=\frac {-2}w\ln x_1.$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.