0
$\begingroup$

Following are two points. lat1, lon1 represent one point and lat2, lon2 represent another point.

lat1, lon1 = 37.317233, -121.94331
lat2, lon2 = 37.317233, -121.94331

When I use the following formula (law of cosines) I get a math domain error

sine_part = sin(lat1_in_radians) * sin(lat2_in_radians)
distance = RADIUS_OF_EARTH_IN_METRES * acos(sine_part + cos(lat1_in_radians) * cos(lat2_in_radians) * cos(lon2_in_radians - lon1_in_radians))

where degrees to radians is converted by decimal_degree / 57.2957795

The weird thing is, the following points work as expected though. I get 0 as my distance.

lat1 = 54.11102, lon1 = -111.12345
lat2 = 54.11102, lon2 = -111.12345

I do understand that the values resulted by applying one of these trignometric functions is not in the interval it could work with. However I would like to know why this error happens in more detail from one of the experts here please.

I'm hoping I haven't done a school boy error, however I have forgotten my school maths so I won't be surprised if I have missed a trivial point!

$\endgroup$
3
  • 2
    $\begingroup$ Do you really want your "two" points to be the same point? $\endgroup$ Commented Apr 11, 2014 at 12:07
  • $\begingroup$ Not really, but I'm curious to know why this happens. $\endgroup$ Commented Apr 11, 2014 at 12:43
  • $\begingroup$ With rounding, the value inside acos() might be fractionally over 1 in your example with the error. I guess it should be exactly 1? $\endgroup$
    – Paul
    Commented Apr 11, 2014 at 13:15

1 Answer 1

1
$\begingroup$

This should fix your code:

sine_part = sin(lat1_in_radians) * sin(lat2_in_radians)
temp = sine_part + cos(lat1_in_radians) * cos(lat2_in_radians) * cos(lon2_in_radians - lon1_in_radians)
if temp > 1:
    temp = 1
if temp < -1:
    temp = -1
distance = RADIUS_OF_EARTH_IN_METRES * acos(temp)

As everyone is telling you in the comments, because of floating point errors, temp, which should be between $-1$ and $1$, might be fractionally outside of this range.

(By the way, I don't know what programming language you are using. But I think the code I wrote is self explanatory.)

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .