Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I am working on cleansing a cities database and trying to increase the precision of the latitudes and longitudes of these cities. I am comparing what I have against an outside datasource and want to determine whether the latitudes and longitudes of any given city are likely more precise in our database or in the outside datasource.

So, for example:

  • 1.54323 is likely more precise than 1.5
  • 1.66591 is likely more precise than 1.6667
  • 1.81839 is likely more precise than 1.818181

Is there a simple mathematical way to determine the likely precision of a decimal number?

Clarified below

Maybe another way to approach this would be ask whether there is a simple way to reverse a decimal expansion.

  • .5 -> 1/2
  • .667 -> 2/3
  • .8181 -> 9/11

If I could get this far, then I could probably establish a rough precision metric simply by summing or multiplying the numerator and denominator.

(There might be a better tag for this. Feel free to add one)

share|improve this question
1  
Other than comparing the number of significant digits and hoping that 1.5 didn't somehow get turned into 1.5000000? No. –  fgp Mar 3 at 23:11
1  
What does "likely more precise" mean? Less chance of having been rounded off at some point? Even if it got rounded off, you never know - the rounded off value might be closer to the real value than the "more precise" measurement. –  Jack M Mar 3 at 23:47
    
@jackm that is why I chose the word "likely". I understand that this is an imperfect approach for solving this problem. I am looking for the number less likely to be a simple fraction –  Scott Rogowski Mar 4 at 0:48

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.