Is the definition of a nautical mile ultimately incorrect?

Question

From many sources the nautical mile is defined as exactly 1852 meters (with minor variances). Based on history it was calculated around one minute of latitude, which is one sixtieth of a degree of latitude.

Now if we calculate a nautical mile based the equitorial radius $ R_e $ of earth, 6378136.6 meters (was the most accurate value I could find)

$$ M = (R_e2\pi)/360/60 $$ which comes to 1855.32473

Is there something im missing ultimately from how the exact definition has become defined because 1852 Meters would in theory not sensibly match up (run short or run over) if an imaginary line was layed around the equator.

Reversing this calculation to calculate the equatorial radius based on the nautical mile being 1852 meters exact we end up with an equatorial radius of 6366707.0194937075

which is a somewhat large difference.

Granted as earth is such a large body and is not a uniform sphere (more of an Ellipsoid with a flatening factor of $ 1 / 298.257223563 $), the amount of error is unpredicatable, but one would hope there is a reasonably accurate value of 1 nautical mile calculated.

Am I over complicating this?

Answer 1

First, a bit of history. In the 16th century the spherical representation of the Earth was generally accepted and began to be used, for navigation and cartographic purposes, with a coordinate system made of latitudes and longitudes. These coordinates are angles (as opposed to linear units), are were reckoned using various instruments that could measure angles of objects in the sky. It was a convenient way for navigators back then to assess their location on the planet, and create better maps of the Earth.

In the 17th century, a British mathematician named Edmund Gunter proposed to use the nautical mile as a convenient unit of distance for navigation, and would correspond to 1/60th of a degree of latitude, or 1 arcminute, because it would be easily relatable to measured angles.

However, one thing that cartographers did not know back then, is that the Earth isn't perfectly spherical. In reality, it more closely resembles an oblate spheroid. As a result, the arcminute of latitude on the Earth's surface isn't equal everywhere. It is slightly longer near the poles, and smaller near the Equator. A navigator moving from 0° 00' to 0° 01'N will obtain a distance of 1,842.9 m, but from 45° 00' to 45° 01'N would obtain 1,852.2 m, and from 89° 59' to 90°N, 1,861.6 m. As a result of this and other possible measurement errors, different countries did not agree on the length of the nautical mile, and several different values were used.

Eventually, as cartographers began to better understand the shape of the Earth, the discrepancy was better explained. Now we know that an arcminute's length on the Earth depends on the location. That new model of the Earth caused the nautical mile to be variable, and since it is highly problematic to have a non-constant length unit, in the end, the nautical mile was agreed internationally to be equal to 1,852 meters, a rounded value in meters close to its length at 45° latitude, also close to 1/5400th of the meridian arc. It is still widely used nowadays, but we know that it only approximately relates to angles.

As a side note, the equatorial radius of the WGS84 ellipsoid, the reference used by GPS systems, is 6,378,137 meters. However, different countries and regions of the world use different ellipsoids that better approximate the local shape of the Earth, and different equatorial radii and flattenings are used. Obtaining a more "accurate" equatorial mean radius and total circumference would be much more involved, because one would need to consider the geoid shape, which deviates by as much as 100 meters from an idealized ellipsoid.

Answer 2

You are holding 18th century measurements to a 20 century standard.

These are supposed to be based on the circumference around a longitude and not around the equator.

In theory, 1 nautical mile corresponds to one arcminute of latitude. There are $5,400$ arcminutes from the equator to the north pole.

Also, in theory it is $10,000,000$ meters from the equator to the north pole (along the Paris meridian).

Since they are based on the same definition it should be exactly $1 \text {M} = \frac {10,000}{5400} \text{m}$

As the earth is oblate, 1 arcminute of latitude is not the same at the poles as it is at the equator. The nautical mile is based on the arcminute at the 45 degree parallel.

Aslo, since in the late 18th century, there was no way to precisely measure the length of the meridian that defined the meter, the French made their best guess based on the technology of the time and created a meter stick that would become the standard meter. It was never exactly $10,000,000 $ meters from the equator to the north pole.

In the 20th century the mile was established as exactly 1852 meters.