# How to compare numbers of different precision

This relates to an argument I'm having with some colleagues, but I believe it is actually a math question.

We have a specification for a material that a certain property be $\leq 0.5$. However our gauge reads to a greater precision than the specification say $0.54$. The obvious solution would be to change the specification, but the argument we're having is whether it is more mathematically consistent to round the reading on the gauge to the nearest tenth, which would indicate that the product is within specification, or to regard the specification as $[0.45, 0.55)$.

Is either method more mathematically sound than the other?

If the second option is correct, does that mean that the result of the comparison is indeterminate?

-
You have a third option: declare 0.54 out of spec. – deinst Feb 10 '12 at 19:50
Or unilaterally declare it in-spec. While this is the most pragmatic approach, it's as 'politically' inconvenient as changing the specification, which is what I've advocated. – Garrett Rowe Feb 10 '12 at 19:57
A $0.54$ diameter rod into a precisely machined $0.51$ diameter hole won't go. Level of precision in deciding between in spec or out of spec is context dependent. – André Nicolas Feb 10 '12 at 20:29
@André: Come on, put on your project management hat. If it actually works with 0.54 (try it first in practice once or twice if you're really paranoid), actually documenting the decision to declare this to be in-spec is just a waste of paper and toner. – Henning Makholm Feb 10 '12 at 20:45
Also: if I were writing specs, and I assume first-digit-past-decimal-point precision, I would not spec a 0.5 rod and a 0.5 hole. I do believe that the convention should be to spec the hole at least 1 unit (in the highest precision digit) bigger than the spec for the rod... – Willie Wong Feb 13 '12 at 16:38

Is either method more mathematically sound than the other?

To your specific question about the difference between "rounding to the nearest tenth" versus "regarding the spec as meaning $[0.45,0.55)$": they mean exactly the same thing mathematically (if you take the convention of rounding to the nearest tenth with halves rounded up), so there's no issue of one being more mathematically sound than the other.

But whether you should interpret your spec as such depends on your engineering convention. Let me just throw in some (mathematically) equally valid ways of interpreting a finite precision spec.

1. You can define $\leq$ to be a partial order on "strings of digits" so that two numbers can only be compared if their decimal representation has the same number of digits past the decimal point. So $0.50 \leq 0.54$. But $0.5$ and $0.54$ are not comparable. Note that in this context, $0.50$ and $0.5$ are not the same object!
2. You can define $\leq$ to be the lexicographical ordering: that is, two strings of digits $A$ and $B$ has $A\leq B$ if the left most digit in which $A$ and $B$ disagree, you have that digit of $A$ is less than that of $B$. And you extend a finite precision number by $0$s. So $0.50 \leq 0.54$. But $0.509 \leq 0.51$.
3. You can define $\leq$ to be the order induced by "rounding". That is, if you have two numbers of different precisions, you declare a "rounding function" $f$ that maps high precision numbers to low precision numbers, so that $A \leq B$ only if $f(A) \leq f(B)$ (where $f(A)$ and $f(B)$ now have the same precision). Of course, in this category you can always round up, always round down, or round to "nearest", or many different rounding functions in between. None is more "mathematically sound" than another.
-

You specification says "less than or equal to $0.5$". In fact $0.54$ is greater than the specification. If it was to be in specification the range would have to be "less than $0.6$"

-