1
$\begingroup$

I am currently coding a program that calls for passing an input value through the sigmoid function, though unfortunately, the software does not accommodate the large number exponentiation that the sigmoid function can entail (ie. -700 as an input would throw an error). Dividing the input by a scalar value (-700/10 for -70) seems the natural way around this, but I don’t want to lose precision. Is there a way to find an ideal, one-size fits all (or most) scalar divisor that minimizes the loss of precision that I can write into the program?

$\endgroup$
0
$\begingroup$

If we are using $S(t)=\frac{1}{1+e^{-t}}$ as the sigmoid function, dividing the exponent $-t$ by some fixed scalar doesn't just have a loss in precision; it yields an incorrect answer that hopefully isn't too wrong. In the case of a large exponent like $-700$, the error is negligible. For an input like $-1$, the error is probably not negligible, depending on your application.

In the context of programming, you have an easier option. Any large enough value of $t$ should evaluate to $1$ in the floating point representation your chosen programming language is using, and any small enough value should evaluate to $0$. Assuming you have around $16$ digits of precision, you would want to solve $\left|1-\frac{1}{1+e^{-t}}\right|<10^{-16}$, which is almost equivalent to $t>16\ln10\approx36.84$.

You could do a similar calculation on the left hand side trying to get $|0-S(t)|<10^{-16}$ and similarly find $t<-16\ln10$. This will be symmetric.

In the context of programming, you often know instead that you have, for example, $53$ bits of accuracy. You would then write $t>53\ln2$ instead of $t>16\ln10$.

As far as what you actually do with those numbers, simply have an if/else structure in your code. For example, in python we could naively write

from math import exp

def sigmoid(t):
    if t > 36:
        return 1.
    elif t < -36:
        return 0.
    return 1/(1+exp(-t))

This would fix any "large exponent" problems you were having.

$\endgroup$
  • $\begingroup$ Thank you for this. If I have to pass multiple inputs through the Sigmoid function for the same run, am I allowed to let my if statement dictate different scalar values for the inputs depending on the conditions of the if statement? It’s weird for me because I think of the whole point of a scalar value as being applied across an entire set of numbers, though I know that the exponent and the properties of the sigmoid function might allow me to use different scalars for the different inputs. $\endgroup$ – pmse234 Jan 4 '18 at 14:15
  • $\begingroup$ It depends what you mean by "multiple inputs ... for the same run." You could feasibly use different scalars for different inputs if you would like, but what you have shown here is convenient because it (probably) doesn't run into any overflow errors (i.e., your exponents are never too big), and it still gives you an exact answer, at least within the limits of your machine's precision. $\endgroup$ – Hans Musgrave Jan 4 '18 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.