Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Short version:

I would like to calculate the expected value if you apply the sigmoid function $\frac{1}{1+e^{-x}}$ to a normal distribution with expected value $\mu$ and standard deviation $\sigma$.

If I'm correct this corresponds to the following integral:

$$\int_{-\infty}^\infty \frac{1}{1+e^{-x}} \frac{1}{\sigma\sqrt{2\pi}}\ e^{ -\frac{(x-\mu)^2}{2\sigma^2} } dx$$

However, I can't solve this integral. I've tried manually, with Maple and with Wolfram|Alpha, but didn't get anywhere.

Some background info (why I want to do this):

Sigmoid functions are used in artificial neural networks as an activation function, mapping a value of $(-\infty,\infty)$ to $(0,1)$. Often this value is used directly in further calculations but sometimes (e.g. in RBM's) it's first stochastically rounded to a 0 or a 1, with the probabililty of a 1 being that value. The stochasticity helps the learning, but is sometimes not desired when you finally use the network. Just using the normal non-stochastic methods on a network that you trained stochastically doesn't work though. It changes the expected result, because (in short):

$$\operatorname{E}[S(X)] \neq S(\operatorname{E}[X])$$

for most X. However, if you approximate X as a normal distribution and could somehow calculate this expected value, you could eliminate most of the bias. That's what I'm trying to do.

share|improve this question
add comment

2 Answers

up vote 2 down vote accepted

I doubt that there's a closed-form solution. However, here's a series in powers of $\sigma$:

$$ \left( {{\rm e}^{-{\mu}}}+1 \right) ^{-1}+{\frac { \left( { {\rm e}^{-{\mu}}}-1 \right) {{\rm e}^{-{\mu}}}}{2\, \left( {{\rm e} ^{-{\mu}}}+1 \right) ^{3}}}{{\sigma}}^{2}+{\frac { \left( { {\rm e}^{-3\,{\mu}}}-11\,{{\rm e}^{-2\,{\mu}}}+11\,{{\rm e}^{-{ \mu}}}-1 \right) {{\rm e}^{-{\mu}}}}{8\, \left( {{\rm e}^{-{\mu} }}+1 \right) ^{5}}}{{\sigma}}^{4}+{\frac {{{\rm e}^{-{\mu} }} \left( {{\rm e}^{-5\,{\mu}}}-57\,{{\rm e}^{-4\,{\mu}}}+302\,{ {\rm e}^{-3\,{\mu}}}-302\,{{\rm e}^{-2\,{\mu}}}+57\,{{\rm e}^{-{ \mu}}}-1 \right) }{48\, \left( {{\rm e}^{-{\mu}}}+1 \right) ^{7}}}{{ \sigma}}^{6}+{\frac {{{\rm e}^{-{\mu}}} \left( {{\rm e}^{-7\,{\mu}}}-247\,{{\rm e}^{-6\,{\mu}}}+4293\,{ {\rm e}^{-5\,{\mu}}}-15619\,{{\rm e}^{-4\,{\mu}}}+15619\,{ {\rm e}^{-3\,{\mu}}}-4293\,{{\rm e}^{-2\,{\mu}}}+247\,{{\rm e}^{ -{\mu}}}-1 \right) }{384\, \left( {{\rm e}^{-{\mu}}}+1 \right) ^{9}}} {{\sigma}}^{8}+O \left( {{\sigma}}^{10} \right) $$

EDIT: To obtain this, first do the change of variables $x = \mu + \sigma t$. The integral becomes $$ \frac{1}{\sqrt{2\pi}} \int_{-\infty}^\infty \dfrac{e^{-t^2/2}}{1 + e^{-\mu - \sigma t}}\ dt $$ Now take the Maclaurin series $$\frac{1}{1+e^{-\mu - \sigma t}} = \frac{1}{1+e^{-\mu}} + \frac{e^{-\mu} \sigma t}{(1+e^{-\mu})^2} + \frac{e^{-\mu} ( e^{-\mu} - 1) \sigma^2 t^2}{(1+e^{-\mu})^3} + \ldots$$ and integrate term by term.

share|improve this answer
1  
Hi Robert, could you comment on the derivation of that series? Where does it come from? –  wvoq Dec 18 '12 at 21:52
add comment

Apart from the the MacLaurin approximation, the usual way to compute that integral in Statistics is to approximate the sigmoid with a profit function. More specifically $\mathrm{sigm}(a) \approx \Phi(\lambda a)$ with $\lambda=\pi/8$. Then the result would be: $$\int \mathrm{sigm}(x) \, N(x \mid \mu,\sigma^2) \, dx \approx \int \Phi(\lambda x) \, N(x \mid \mu,\sigma^2) \, dx = \Phi\left(\frac{x}{\sqrt{\lambda^{-2} + \sigma^2}}\right).$$

share|improve this answer
    
Unless I'm mistaken somewhere, $\lambda = \pi / 8$ isn't a particularly good value of $\lambda$ to use. The best value, in terms of max-norm convergence, seems to be about 0.588 (about $\pi / 5.35$). –  Dougal Jan 19 at 19:43
    
I should also note for posterity that, since the equality wasn't obvious to me, here's a nice proof. –  Dougal Jan 19 at 20:38
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.