Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I made the following observation following a suggestion from a friend. This question is about the Microsoft calculator program implemented on Windows. Since I don't have access to any other calculator at the moment, I cannot confirm whether this is a general feature.

If you enter an integer in the calculator program, say $N$, hit the minus button followed by the same integer again, you get zero as expected.

However, if you input $N^2$, then hit square root button, you get $N$, then subtract $N$, you get a very small number, but not zero.

This has been my observation for several integers that I have tried.

For example, with $N=5$, you get $1.232\times 10^{-24}$ in the final step.

Now, I understand computers use finite precision (and hence the square root function must be working under finite precision). However, I don't understand why this shows up in the final step only? Is it only because this is a small number in comparison to any integer, but when you get $0$, you can no longer make that approximation?

share|cite|improve this question
I suspect a very poor approximation to the square root is being used (but since the software is closed-source, your guess is as good as mine). Could you try in Excel? – J. M. Nov 26 '10 at 4:50
@J.M.: Excel is the benchmark! – AD. Nov 26 '10 at 5:42
In case the gag was missed: Excel is also a floating-point headache. :P I really want to hear about more stories of stuff being ruined by sucky floating point arithmetic... – J. M. Nov 26 '10 at 5:49
@J.M.: I never saved it, but I had an algebraic expression with plus, minus, mult and div in that was supposed to add up to 1 but in Excel (latest) it did not.. – AD. Nov 26 '10 at 6:02
I agree. It is not hard to implement arithmetic using strings. – AD. Nov 26 '10 at 6:04
up vote 8 down vote accepted

You'll find your answer and much more in the articles below:

What Every Computer Scientist Should Know About Floating-Point Arithmetic

Also see this article on Calculator Forensics, esp. "Hints and Tricks For Obtaining Guard Digits".

share|cite|improve this answer
+1 for the beautiful article, which should be read after perusing the IEEE guidelines. Velvel Kahan's miscellaneous papers in his site, like this one are also an excellent read. – J. M. Nov 26 '10 at 4:54

J.M., since when is 24 decimal digits accuracy considered a bad approximation?

Original poster, they could show sqrt(5)**2 as


but since the display probably has fewer display digits than this, they chop it to "5" as that is the closest representation in the allowed number of digits.

BTW, I just tried this in "calc" in windows 7 and the 5-sqrt(5)**2 is 1.7599e-37

If you want exact math, you will need to use a symbolic math package.

share|cite|improve this answer
There is such a thing as a properly rounded square root (most certainly if the routine is to handle (near-)integer inputs). You might want to look at the link I gave in the comment to Bill's answer. – J. M. Nov 26 '10 at 5:58
First, my apologies for posting what I intended to be a comment as an answer, which it isn't. – Jim B Nov 26 '10 at 7:10

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.