0
$\begingroup$

I'm working on an algorithm that gets the weighted least squares of some data and outputs really big, or really small numbers. I'm talking about numbers like $3.55114473577E+256$.

Now, part of the algorithm I'm coding needs to use these numbers and raise $e$ to their power, in the following formula:

$β = e^x / (1 + e^x) $

Where β values will then be used as weights for regression and such.

But $e^{3.55114473577e+256}$ is pretty impossible.

So in such problems, what's the best solution? What kind of rescaling should be done to the numbers I'm getting? Or, alternatively, what's the logarithmic equivalent to the above equation? Because I'm aware that to deal with gigantic numbers, logarithms can be used instead of exponents.

$\endgroup$
2
$\begingroup$

For large $x$, $\frac{e^x}{1+e^x}$ is very close to $1$. You have $$ 1 - \frac{e^x}{1 + e^x} = \frac{1}{1 + e^x} $$ so the relative error of assuming $\frac{e^x}{1+e^x} = 1$ is about $e^x$. That is so insanely tiny for $x \approx 10^{256}$ that I very much doubt that it matters. If your algorithm really depends on $\beta$, i.e. if you cannot just replace it with $1$, then I doubt that what you're trying to do is numerically feasable.


More generally, the function $f(x) = \frac{e^x}{1 + e^x}$ has the following behaviour

  1. $f$ is a strictly monotonic function from $\mathbb{R}$ to $(0,1)$.
  2. $f(0) = \frac{1}{2}$, $f(x) \to 1$ as $x \to \infty$, $f(x) \to 0$ as $x \to -\infty$.
  3. For $x \geq 10^n$, $f(x) \geq 1 - 10^{-\left(10^{n-1}\right)}$.
  4. For $x \leq -10^n$, $f(x) \leq 10^{-\left(10^{n-1}\right)}$

(3) and (4) should give you an idea - depending on the precision of your numbers - how large $x$ must be to safely replace $\frac{e^x}{1 + e^x}$ with $1$. enter image description here

$\endgroup$
  • $\begingroup$ So does that mean that the functional values of expression $e^x / (1 + e^x)$ are likely to either be 0, or 1, or anywhere in between? $\endgroup$ – user961627 May 7 '14 at 16:13
  • $\begingroup$ @user961627 That depends on $x$, see my edit $\endgroup$ – fgp May 7 '14 at 16:39
  • $\begingroup$ One more question - this function - the logistic function, is used for "normalization", right? $\endgroup$ – user961627 May 8 '14 at 11:31
  • $\begingroup$ @user961627 Well, it compresses the whole real line into $(0,1)$, and acts almost linearly on values with a small enough absolute values. So yes, you can use it to normalize values that might get arbitrarily large into $(0,1)$. But without further context, "normalization" is a rather broad term, so whether this applies to your context is hard to say. If you require further information, I think you should explain what you're actually trying to do in your question... $\endgroup$ – fgp May 8 '14 at 11:47
  • $\begingroup$ This is the formula I'm actually working with: i.stack.imgur.com/f65Na.png As you can see, if we simplify all the $g_i(stuff)$ and just call it $g$, then this formula isn't exactly $e^x / (1 + e^x)$. It's more like $e^x / (1 + sum(different e^{g(x)s})$. And I'm thinking this isn't serving to properly normalize? Because it's possible that the $g(x)s$ end up negative... and so we might end up with something like $e^{-2} / ( 1 + e^{large number} )$, which may therefore produce a huge number...Am I right? I need the β values as coefficients to build weights for weighted least sq regression. $\endgroup$ – user961627 May 8 '14 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.