2
$\begingroup$

I have tried this on my Windows 7 calculator with $\sqrt9 -3 $ it too gives some weird answer- ie$1.1546388020691628168216106791278e-37$. And so for any $n$(positive) $\sqrt n^2-n= wierd_ .answer$
Why does this happen?

$\endgroup$
3
$\begingroup$

Presumable, the Windows 7 calculator uses floating point arithmetic to do these computations.

The IEEE 754 floating point standard requires that basic arithmetic operations (which include sqrt) are correctly rounded: the result should be as if it was computed with infinite precision and then correctly rounded to fit in a double.

Apparently, the Windows 7 calculator has a bug: instead of the correct answer 2 (which is of course exactly representable in a double) it seems to come up with the largest double smaller than 2. If you then subtract 2, you get something very close to but not exactly equal to 0.

$\endgroup$
  • 1
    $\begingroup$ I don't think it is a bug, roots are calculated afaik via iterations so the errors may grow extremely. If you have double floating point precision, shouldn't the largest machine number smaller than 2 be something like $2-10^{-16}$ and not something like $2-10^{-39}$ ? $\endgroup$ – Dominic Michaelis Oct 6 '13 at 16:02
  • 1
    $\begingroup$ I understand why it happens, but it's still wrong - it does not meet the specification. The makers probably didn't care and don't consider it a bug, though, but there I'm guessing. As for the the $2 - 10^{-16}$ vs. $2 - 10^{-39}$, it's probably extended double instead of double (128 bits instead of 64), but I didn't check. $\endgroup$ – Magdiragdag Oct 6 '13 at 16:05
1
$\begingroup$

The weird answer $-8.16\ldots e-39$ stands for $-8.16\ldots \cdot 10^{-39}$. It happens because a calculator just calculates numerical, so the maximal precision is the so called machine precision, which is usally something about $10^{-16}$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.