Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I have tried this on my Windows 7 calculator with $\sqrt9 -3 $ it too gives some weird answer- ie$1.1546388020691628168216106791278e-37$. And so for any $n$(positive) $\sqrt n^2-n= wierd_ .answer$
Why does this happen?

share|improve this question
2  
    
Floats only store up to 8 bits of data iirc –  Don Larynx Oct 6 '13 at 15:46
    
So your calculator is close, but no cigar. Thus use a more reliant calculator. –  Michael Hoppe Oct 6 '13 at 15:47

2 Answers 2

up vote 3 down vote accepted

Presumable, the Windows 7 calculator uses floating point arithmetic to do these computations.

The IEEE 754 floating point standard requires that basic arithmetic operations (which include sqrt) are correctly rounded: the result should be as if it was computed with infinite precision and then correctly rounded to fit in a double.

Apparently, the Windows 7 calculator has a bug: instead of the correct answer 2 (which is of course exactly representable in a double) it seems to come up with the largest double smaller than 2. If you then subtract 2, you get something very close to but not exactly equal to 0.

share|improve this answer
1  
I don't think it is a bug, roots are calculated afaik via iterations so the errors may grow extremely. If you have double floating point precision, shouldn't the largest machine number smaller than 2 be something like $2-10^{-16}$ and not something like $2-10^{-39}$ ? –  Dominic Michaelis Oct 6 '13 at 16:02
1  
I understand why it happens, but it's still wrong - it does not meet the specification. The makers probably didn't care and don't consider it a bug, though, but there I'm guessing. As for the the $2 - 10^{-16}$ vs. $2 - 10^{-39}$, it's probably extended double instead of double (128 bits instead of 64), but I didn't check. –  Magdiragdag Oct 6 '13 at 16:05

The weird answer $-8.16\ldots e-39$ stands for $-8.16\ldots \cdot 10^{-39}$. It happens because a calculator just calculates numerical, so the maximal precision is the so called machine precision, which is usally something about $10^{-16}$

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.