# Rounding trouble in a calculator's square root function

I made the following observation following a suggestion from a friend. This question is about the Microsoft calculator program implemented on Windows. Since I don't have access to any other calculator at the moment, I cannot confirm whether this is a general feature.

If you enter an integer in the calculator program, say $N$, hit the minus button followed by the same integer again, you get zero as expected.

However, if you input $N^2$, then hit square root button, you get $N$, then subtract $N$, you get a very small number, but not zero.

This has been my observation for several integers that I have tried.

For example, with $N=5$, you get $1.232\times 10^{-24}$ in the final step.

Now, I understand computers use finite precision (and hence the square root function must be working under finite precision). However, I don't understand why this shows up in the final step only? Is it only because this is a small number in comparison to any integer, but when you get $0$, you can no longer make that approximation?

-
I suspect a very poor approximation to the square root is being used (but since the software is closed-source, your guess is as good as mine). Could you try in Excel? –  Guess who it is. Nov 26 '10 at 4:50
@J.M.: Excel is the benchmark! –  AD. Nov 26 '10 at 5:42
In case the gag was missed: Excel is also a floating-point headache. :P I really want to hear about more stories of stuff being ruined by sucky floating point arithmetic... –  Guess who it is. Nov 26 '10 at 5:49
@J.M.: I never saved it, but I had an algebraic expression with plus, minus, mult and div in that was supposed to add up to 1 but in Excel (latest) it did not.. –  AD. Nov 26 '10 at 6:02
I agree. It is not hard to implement arithmetic using strings. –  AD. Nov 26 '10 at 6:04

What Every Computer Scientist Should Know About Floating-Point Arithmetic

Also see this article on Calculator Forensics, esp. "Hints and Tricks For Obtaining Guard Digits".

-
+1 for the beautiful article, which should be read after perusing the IEEE guidelines. Velvel Kahan's miscellaneous papers in his site, like this one are also an excellent read. –  Guess who it is. Nov 26 '10 at 4:54

J.M., since when is 24 decimal digits accuracy considered a bad approximation?

Original poster, they could show sqrt(5)**2 as

5.000000000000000000000001232

but since the display probably has fewer display digits than this, they chop it to "5" as that is the closest representation in the allowed number of digits.

BTW, I just tried this in "calc" in windows 7 and the 5-sqrt(5)**2 is 1.7599e-37

If you want exact math, you will need to use a symbolic math package.

-
There is such a thing as a properly rounded square root (most certainly if the routine is to handle (near-)integer inputs). You might want to look at the link I gave in the comment to Bill's answer. –  Guess who it is. Nov 26 '10 at 5:58
First, my apologies for posting what I intended to be a comment as an answer, which it isn't. –  Jim B Nov 26 '10 at 7:10