# What are the units of $R$ in $(x−x_0)^2+(y−y_0)^2=R^2$ if coordinates are latitude and longitude?

I want to use this formula $(x−x_0)^2+(y−y_0)^2=R^2$ to determine if $(x, y)$ point is within circle with center $(x_0, y_0)$ and radius $R$.

But, in what units is $R$ specified, if coordinates are a decimal geo-coords (latitude, longitude)? And how can I convert radius to miles or kilometers and vice-versa?

Thank you!

• That formula only works in a flat Cartesian coordinate system. It is not meant to be used with latitude and longitude on the curved surface of the Earth. – David K Apr 18 '18 at 16:50
• @DavidK any help is appreciated to modify formula to fit requirements. – Andrey Prokhorov Apr 18 '18 at 16:54
• Cannot be done with the information you provided. Let's say you have a circle with constant latitude (for example the equator equation is latitude=0). The same equation applies to different sphere sizes. So the equator on the Moon has the same equation as the equator on the Earth, or the equator on the Sun. But the radius is completely different. – Andrei Apr 18 '18 at 16:58
• @Andrei I'm going to use formula to determine if aircraft is within some range from the airport. I know aircraft and airport coordinates. Aircraft altitude is not taken in account. – Andrey Prokhorov Apr 18 '18 at 17:00
• All you need to do is to calculate the distance from aircraft to the airport and compare it to radius $R$. See, for instance, movable-type.co.uk/scripts/latlong.html – Vasya Apr 18 '18 at 17:05

## 1 Answer

Ok, including all valuable mentions in comments I went to solution. We can use this formula to determine distance between two points specified by lat-lon coordinates:

$d = \arccos\bigl(\sin lat_x * \sin lat_0 + \cos lat_x * \cos lat_0 * cos(lon_x - lon_0)\bigr)$

where $(lat_x, lon_x)$ is an aircraft coords in radians, and $(lat_0, lon_0)$ - airport coordinates in radians, $d$ - distance in radians.

To convert decimal coordinates to radians we need to do next:

$r = \frac{dec * \pi}{180}$,

where $r$ - lat or lon in radians, $dec$ - decimal value of lat or lon.

After that we can convert distance in radians into nautical miles (nm):

$d_{nm}= \frac{180*60*d}{\pi}$

or into kilometers:

$d_{km} = FAI * d$,

where FAI is equal to 6371 km (if we assume that earth is a perfect sphere).

• The formula for distance can simply be stated as $d_{\mathrm units} = FAI_{\mathrm units} \times d_{\mathrm radians},$ that is, if you multiply the output of $\arccos$ by the radius of the Earth in whatever units you want (km, nm, inches, parsecs, or whatever), the product will be the distance from the airport in the same units. Note that the generally accepted mean Earth radius in nm is very slightly greater than $180\times60/\pi,$ although it is less than $0.1\%$ greater so perhaps the difference doesn't matter to you. – David K Apr 18 '18 at 22:20