1
$\begingroup$

I want to find the solution to $\sigma(x) =\sigma^{-1}(x)$ Where $\sigma(x)=\frac{1}{1+e^{-x}}$ and so $\sigma^{-1}(x)=\ln(x)-\ln(1-x)$. I've got it down to $\frac{e^x}{x}=e^x+1$ but I can't get any further. Desmos tells me that the solution is around $0.659$ but I want an exact expression for it... Is this possible?

$\endgroup$
  • $\begingroup$ I don't believe an analytic closed form solution exists. $\endgroup$ – superckl Nov 13 '17 at 19:18
  • $\begingroup$ @mathreadler nothing in particular, just trying to get a feel for the sigmoid function for working with neural networks $\endgroup$ – Morgan Saville Nov 13 '17 at 19:21
  • $\begingroup$ Okay, thank you :)) @superckl $\endgroup$ – Morgan Saville Nov 13 '17 at 19:21
  • $\begingroup$ Maybe Lamberts W function could help you express the solution. I dont see how this particular equation gives any insight into neural networks. Why don't you build some intuition for the multivariate chain rule instead. $\endgroup$ – mathreadler Nov 13 '17 at 19:23
  • 1
    $\begingroup$ The equation that you derived is simply $\sigma(x) =x$. But that's not the original. $\endgroup$ – Friedrich Philipp Nov 13 '17 at 19:24
1
$\begingroup$

I don't believe your equation has an analytic solution. You can, however, get close by trying some approximations. Let's rewrite our equation as

$$e^x=\frac{x}{1-x}$$

Say we expand the functions into a Taylor series

$$e^x=1+x+\frac{x^2}{2}+...$$ $$\frac{x}{1-x}=x-x^2+...$$

Dropping higher order terms, we have

$$1+x+\frac{x^2}{2}=x+x^2$$

which, when simplified, gives us

$$x^2-2=0$$

Which tells us $x=\pm\sqrt{2}\approx1.414$

Okay not great... but adding another higher order term gives us

$$5x^3+3x^2-6=0$$

Which has one real root at $x\approx.895$. Adding the fourth order term gives $x\approx.767$. A fifth gives $x\approx.716$.

By the time you've reached seventh order terms, your approximation is at $x\approx.679$ which is within 3% of the actual answer. This is just one way to get at the solution by using functions we know the solutions to.

$\endgroup$
  • $\begingroup$ Well, "by using functions we know the solutions to" is not quite correct. I mean, determining the roots of a 7th order polynomial can also only be done numerically. $\endgroup$ – Friedrich Philipp Nov 13 '17 at 19:39
  • $\begingroup$ I suppose that's correct, I meant they are more "familiar" in a sense. $\endgroup$ – superckl Nov 13 '17 at 19:43
  • 1
    $\begingroup$ Polynomials might be more familiar. However, this doesn't give you nothing for retrieving the roots. Also, polynomials have several roots, while the given function $\sigma(x)-x$ only has a single root. I think Newton's method works better here. $\endgroup$ – Friedrich Philipp Nov 13 '17 at 19:49
  • $\begingroup$ I would agree, I've never claimed this method is better. It's simply an alternative. $\endgroup$ – superckl Nov 13 '17 at 19:52
0
$\begingroup$

Use the second order approximation of $\textbf{exp}$ i.e $e^x \approx 1+x+\frac{x^2}{2!}$. We would then have,

$$\frac{e^x}{x} = e^x + 1 \Rightarrow \frac{1+x+\frac{x^2}{2!}}{x} \approx \left(1+ x+ \frac{x^2}{2!}\right)+1$$

The above implies,

$$1+ x + \frac{x^2}{2!} \approx 2x + x^2 + \frac{x^3}{2!} $$

And so $x$ will be approximately the root of,

$$ p(u) =u^3 + u^2 + 2u-2 $$

Now you can use intermediate value theorem first noting that $p(.5)<0$ and $p(1)>0$ i.e your zero lies in the interval $[.5,1]$. If you want to approach this from a qualitative point of view. I would just keep increasing $.5$ until $p(.5+ \epsilon)>0$. In any case, at least now you can an expression which approximates the root.

$\endgroup$
0
$\begingroup$

At first, the value approximated by your computation system is correct. Using algebraic adjustments we can determine an equality: $$-\log\left({\frac{1}{x}-1}\right)= \frac{e^x}{e^x+1}$$ The only possible way to evaluate the solution to this equation is explicitly numeric, therefore indescribable as an analytical expression. The closest thing to get an explicit number I can think of is a convergence of Fourier or Taylor series, which can be described as an infinite sum of functions. There should not exist any way to do this precisely and nothing should be more precise considering absolute convergence than those series.

$\endgroup$
0
$\begingroup$

To find a root of $g(x)=(x-1)e^x+x$ in $[0,1]$ is numerically pretty simple since such function is convex and increasing over $[0,1]$. By initializing Newton's method $$ z_{n+1} = z_n - \frac{g(z_n)}{g'(z_n)} $$ at $z_0=1$ we get $z_2<\frac{2}{3}$ and $z_4=\color{green}{0.659046}\ldots$ You might have an explicit representation of such constant in terms of Lambert $W$ function, but the computation of $W$ still is performed through Newton's method, raising an interesting debate about what explicit really means.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.