2
$\begingroup$

Let $\sigma(z) = 1/(1+e^{-z})$. How can I find $w_1, w_2, b$ such that $w_2 \sigma(w_1 x + b) \approx x$ for $x \in [0,1]$?

The hint provided was to rewrite $x = \frac{1}{2}+\Delta$, assume $w_1$ is small, and use a Taylor expansion in $w_1 \Delta$.

(This problem comes from chapter 5 of Neural Networks and Deep Learning.)

Attempt: Performing the substitution from the hint yields

$$ w_2 \sigma\left(\frac{1}{2} w_1 + w_1 \Delta + b\right) \approx \frac{1}{2} + \Delta. $$

Assuming $w_1$ is small,

$$ w_2 \sigma(w_1 \Delta + b) \approx \frac{1}{2} + \Delta. $$

Taking a first order Taylor expansion of the LHS about $w_1 \Delta = 0$ gives

$$ w_2 \sigma(b) + w_1 w_2 \sigma'(b) \Delta \approx \frac{1}{2} + \Delta, $$

so $w_2 \sigma(b) \approx \frac{1}{2}$ and $w_1 w_2 \sigma'(b) \approx 1$, thus $w_2 \approx \frac{1}{2 \sigma(b)}$ and $w_1 \approx \frac{2 \sigma(b)}{\sigma'(b)}$. However, when I tried a few values of $b$, none of them produced very good approximations of $x$.

For what it's worth, the parameters $w_1=3.98, w_2=1.14, b=-2.27$ work pretty well (found via a grid search).

$\endgroup$
  • 1
    $\begingroup$ Thanks for posting this nice problem ! I had fun. Cheers :-) $\endgroup$ – Claude Leibovici Dec 21 '15 at 9:05
1
$\begingroup$

I think that there is a problem when it is assumed that $w_1$ is small.

Let me show what I tried : developing as a Taylor series around $x=\frac 12$ gives $$w_2 \sigma (w_1 x+b)=w_2 \sigma \left(\frac{w_1}{2}+b\right)+w_1 w_2 \sigma '\left(\frac{w_1}{2}+b\right)\left(x-\frac{1}{2}\right)+O\left(\left(x-\frac{1}{2}\right)^2\right) $$ So, ignoring the higher order terms and rearranging $$w_2 \sigma (w_1 x+b)\approx -\frac{1}{2} w_1 w_2 \sigma '\left(\frac{w_1}{2}+b\right)+w_2 \sigma \left(\frac{w_1}{2}+b\right)+w_1 w_2 \sigma '\left(\frac{w_1}{2}+b\right)x$$ So, you want $$w_1 w_2 \sigma '\left(\frac{w_1}{2}+b\right)=1$$ $$-\frac{1}{2} w_1 w_2 \sigma '\left(\frac{w_1}{2}+b\right)+w_2 \sigma \left(\frac{w_1}{2}+b\right)=0$$ So, the first equation gives $w_2$ as a function of $b$ and $w_1$ and the second equation (in which, by magics, $w_2$ disappears) gives an implicit relation between $w_1$ and $b$. For sure, if we decide to neglect the terms $\frac {w_1} 2$ in the functions, we arrive to your results.

If now, we introduce the definition of $\sigma(z)$, after simplifications, the equations write $$2 e^{\frac{w_1}{2}+b}-w_1+2=0$$ $$\frac{w_1 w_2 e^{-\frac{w_1}{2}-b}}{\left(e^{-\frac{w_1}{2}-b}+1\right)^2}-1=0$$

As functions of $w_1$, the solutions are $$w_2=\frac{1}{w_1-2}+\frac{1}{2}$$ $$b=\log \left(\frac{w_1-2}{2}\right)-\frac{w_1}{2}$$

If we use the above expressions and push the Taylor expansion to higher orders, we obtain $$\frac{1}{2}+\left(x-\frac{1}{2}\right)+\left(2-\frac{w_1}{2}\right) \left(x-\frac{1}{2}\right)^2+O\left(\left(x-\frac{1}{2}\right)^3\right)$$ from which it is clear that $w_1=4$ is the best choice to which correspond $w_2=1$ and $b=-2$ which are very close to the solution you gave.

Edit

The problem can be solved in a simpler manner without using Taylor; considering the function $$f(x)=\frac{w_2}{1+e^{-w_1 x-b}}$$ Basically what is required is to satisfy $$f(\frac 12)=\frac 12 \quad , \quad f'(\frac 12)=1 \quad , \quad f''(\frac 12)=0$$ Here again, the first and second equations allow to eliminate two of the unknowns as a function of the third unknown; the last equation gives the last parameter. For sure, the same results are obtained.

$\endgroup$
  • $\begingroup$ The technique in the edit is very nice! Thank you for showing that! $\endgroup$ – Snowball Dec 22 '15 at 6:12
  • $\begingroup$ @Snowball. Last night, I was thinking that a simpler solution could exist. If you think more about it, it is exactly equivalent to the Taylor expansion but much simpler (you just impose what should be the expansion around $x=\frac 12$. I am glad you like it ! $\endgroup$ – Claude Leibovici Dec 22 '15 at 6:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.