Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

There's an example in my textbook about cancellation error that I'm not totally getting. It says that with a $5$ digit decimal arithmetic, $100001$ cannot be represented.

I think that's because when you try to represent it you get $1*10^5$, which is $100000$. However it goes on to say that when $100001$ is represented in this floating point system (when it's either chopped or rounded) it comes to $100000$.

If what I said above is correct, does $100001$ go to $100000$ because of the fact that it can only be represented like $1*10^5$?

If I'm completely off the mark, clarification would be great.

share|cite|improve this question
up vote 2 down vote accepted

Yes, you only have five decimal digits available. $100001=1.00001*10^5$ but I have six digits in the mantissa. Clearly it is closer to go to $1.0000$ than to $1.0001$, so that is what we will do. So the numbers around here that can be represented are $99998, 99999, 100000, 100010, 100020,$ etc.

share|cite|improve this answer
Also on another note would the largest number be 9.9999 * 10^(whatever exp was max)? – wzsun Feb 13 '13 at 1:23
@wzsun: that is correct. – Ross Millikan Feb 13 '13 at 1:29

I think the point is about the precision of the number that can be stored, not so much the exponent. In other words you cannot store $1.00001*10^5$ because it cannot store five decimal places of precision. Equally it would not be able to store $1.00001*10^{15}$ or $1.00001*10^{-5}$.

share|cite|improve this answer

That is correct. You have what is called type overflow. Want to be amused? If you have a C compiler run this program

int main(void)
    int x = 1;
    while (x > 0)
        x = x + 1;
    printf("x = %d\n", x);

It's educational. This is in integer-world so it's simpler than floating point numbers, but these have an analogous problem: loss of precision. I recommend Forman Acton's Real Computing Made Real for a discussion of this issue.

share|cite|improve this answer
I don't recall my C very well, but why wouldn't the while fail the first trip through? It also seems like if your compiler defaults to 32 bit ints you might have a bit of a wait. Should it print x=-1? – Ross Millikan Feb 13 '13 at 1:28
Note correction. Sorry about the error. – ncmathsadist Feb 13 '13 at 1:38
Integer overflow is a very different situation than floating-point loss of precision. – Nate Eldredge Feb 13 '13 at 1:47
It is a similar problem. The bits run off the end and you get in trouble. – ncmathsadist Feb 13 '13 at 2:03
Go look at how IEEE 754 works. – ncmathsadist Feb 13 '13 at 2:04

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.