# Finding $w_1,w_2,b$ such that $w_2 \sigma(w_1 x + b) \approx x$

Let $\sigma(z) = 1/(1+e^{-z})$. How can I find $w_1, w_2, b$ such that $w_2 \sigma(w_1 x + b) \approx x$ for $x \in [0,1]$?

The hint provided was to rewrite $x = \frac{1}{2}+\Delta$, assume $w_1$ is small, and use a Taylor expansion in $w_1 \Delta$.

(This problem comes from chapter 5 of Neural Networks and Deep Learning.)

Attempt: Performing the substitution from the hint yields

$$w_2 \sigma\left(\frac{1}{2} w_1 + w_1 \Delta + b\right) \approx \frac{1}{2} + \Delta.$$

Assuming $w_1$ is small,

$$w_2 \sigma(w_1 \Delta + b) \approx \frac{1}{2} + \Delta.$$

Taking a first order Taylor expansion of the LHS about $w_1 \Delta = 0$ gives

$$w_2 \sigma(b) + w_1 w_2 \sigma'(b) \Delta \approx \frac{1}{2} + \Delta,$$

so $w_2 \sigma(b) \approx \frac{1}{2}$ and $w_1 w_2 \sigma'(b) \approx 1$, thus $w_2 \approx \frac{1}{2 \sigma(b)}$ and $w_1 \approx \frac{2 \sigma(b)}{\sigma'(b)}$. However, when I tried a few values of $b$, none of them produced very good approximations of $x$.

For what it's worth, the parameters $w_1=3.98, w_2=1.14, b=-2.27$ work pretty well (found via a grid search).

• Thanks for posting this nice problem ! I had fun. Cheers :-) – Claude Leibovici Dec 21 '15 at 9:05

## 1 Answer

I think that there is a problem when it is assumed that $w_1$ is small.

Let me show what I tried : developing as a Taylor series around $x=\frac 12$ gives $$w_2 \sigma (w_1 x+b)=w_2 \sigma \left(\frac{w_1}{2}+b\right)+w_1 w_2 \sigma '\left(\frac{w_1}{2}+b\right)\left(x-\frac{1}{2}\right)+O\left(\left(x-\frac{1}{2}\right)^2\right)$$ So, ignoring the higher order terms and rearranging $$w_2 \sigma (w_1 x+b)\approx -\frac{1}{2} w_1 w_2 \sigma '\left(\frac{w_1}{2}+b\right)+w_2 \sigma \left(\frac{w_1}{2}+b\right)+w_1 w_2 \sigma '\left(\frac{w_1}{2}+b\right)x$$ So, you want $$w_1 w_2 \sigma '\left(\frac{w_1}{2}+b\right)=1$$ $$-\frac{1}{2} w_1 w_2 \sigma '\left(\frac{w_1}{2}+b\right)+w_2 \sigma \left(\frac{w_1}{2}+b\right)=0$$ So, the first equation gives $w_2$ as a function of $b$ and $w_1$ and the second equation (in which, by magics, $w_2$ disappears) gives an implicit relation between $w_1$ and $b$. For sure, if we decide to neglect the terms $\frac {w_1} 2$ in the functions, we arrive to your results.

If now, we introduce the definition of $\sigma(z)$, after simplifications, the equations write $$2 e^{\frac{w_1}{2}+b}-w_1+2=0$$ $$\frac{w_1 w_2 e^{-\frac{w_1}{2}-b}}{\left(e^{-\frac{w_1}{2}-b}+1\right)^2}-1=0$$

As functions of $w_1$, the solutions are $$w_2=\frac{1}{w_1-2}+\frac{1}{2}$$ $$b=\log \left(\frac{w_1-2}{2}\right)-\frac{w_1}{2}$$

If we use the above expressions and push the Taylor expansion to higher orders, we obtain $$\frac{1}{2}+\left(x-\frac{1}{2}\right)+\left(2-\frac{w_1}{2}\right) \left(x-\frac{1}{2}\right)^2+O\left(\left(x-\frac{1}{2}\right)^3\right)$$ from which it is clear that $w_1=4$ is the best choice to which correspond $w_2=1$ and $b=-2$ which are very close to the solution you gave.

Edit

The problem can be solved in a simpler manner without using Taylor; considering the function $$f(x)=\frac{w_2}{1+e^{-w_1 x-b}}$$ Basically what is required is to satisfy $$f(\frac 12)=\frac 12 \quad , \quad f'(\frac 12)=1 \quad , \quad f''(\frac 12)=0$$ Here again, the first and second equations allow to eliminate two of the unknowns as a function of the third unknown; the last equation gives the last parameter. For sure, the same results are obtained.

• The technique in the edit is very nice! Thank you for showing that! – Snowball Dec 22 '15 at 6:12
• @Snowball. Last night, I was thinking that a simpler solution could exist. If you think more about it, it is exactly equivalent to the Taylor expansion but much simpler (you just impose what should be the expansion around $x=\frac 12$. I am glad you like it ! – Claude Leibovici Dec 22 '15 at 6:16