Measuring Distance in Coordinate Systems

In GIS software, distance measurement is a rigorous and complex task. In the real world, we can directly measure true ground distance with a rangefinder. In GIS, distance measurement depends on a coordinate reference system. However, Earth is not a regular sphere but an irregular ellipsoid. Digitizing it as a coordinate system introduces various approximation errors, so distances measured in different coordinate systems may not be exactly the same. Distance calculations in any coordinate system must account for geometric deformation and projection error.

In a geographic coordinate system (also called a geodetic coordinate system), Earth is usually simplified as a reference ellipsoid, a mathematical model used to approximate Earth's shape. Because different ellipsoids use different parameters, such as semi-major axis and flattening, spatial distances calculated from those ellipsoids can also differ, affecting measurement accuracy.

In a projected coordinate system, distance errors are usually more significant. Projecting an ellipsoidal surface onto a two-dimensional plane is itself a deformation process and inevitably introduces distortion. As the measurement extent increases, especially at high latitudes or far from the central meridian, these errors can be amplified further, causing large differences between real-world distance and projected distance.

Therefore, when using GIS software to measure distance, you should fully consider the type of coordinate system in use and the geometric characteristics it introduces to ensure reasonable and accurate results. The software provides four distance options: geodesic, planar, rhumb line, and great ellipse line.

Geodesic Line

A geodesic line is the shortest curved path between two points on Earth's reference ellipsoid. It is essentially the "straight line" on the ellipsoidal surface. On a sphere, this corresponds to a great-circle route; on an ellipsoid, Earth's flattening must also be considered. The direction along a geodesic changes continuously, except along a meridian or the equator, but the total length is the shortest. Geodesics are widely used in surveying, mapping, and navigation, and distance calculations can reach millimeter-level accuracy.

Use Cases

Geodesics are widely used in surveying and mapping, aviation, marine navigation, and military applications. For example, surveying often requires precise distances between two known coordinate points or boundary delineation. Ellipsoidal geodesics are commonly used to ensure accuracy; many international borders and maritime boundaries are formed by connecting endpoint coordinates with geodesics.

In aviation, long-haul flight planning usually uses geodesics, approximated as spherical great circles, to determine the shortest flight path and save fuel. For example, a route from Beijing to New York planned along a geodesic is several percent shorter than one flown on a constant bearing.

In satellite applications, although satellite orbits do not follow Earth's surface, geodesics or great circles are often used as approximations when representing the satellite ground track.

In military applications, a missile's maximum range is usually evaluated as a geodesic distance along Earth's ellipsoidal surface to estimate strike coverage.

Overall, because geodesics represent shortest paths, they are the fundamental choice for GIS analysis and navigation planning that require optimal distance calculations.

Calculation Methods and Algorithms

• Vincenty's formulae: The Vincenty algorithm was proposed by Polish geodesist Thaddeus Vincenty in 1975 to calculate geodesic distance and azimuth between two points on an ellipsoid. It uses an iterative method to solve the inverse geodesic problem, where two latitude/longitude points are known and the shortest distance and azimuth are required, and the direct geodesic problem, where the start point, azimuth, and distance are known and the endpoint is required. Vincenty's formulae are based on reference ellipsoid parameters such as the semi-major axis and flattening of WGS 84. The algorithm converts geodetic latitude to reduced latitude on an auxiliary sphere, applies spherical trigonometry, and then iteratively corrects the distance. This method is more complex than a simple spherical great-circle formula but is highly accurate, reaching about 0.5 mm on Earth's ellipsoid. Note that Vincenty's algorithm may fail to converge when the two points are near antipodal, meaning almost opposite sides of Earth.

• Karney algorithm (GeographicLib): To improve robustness and accuracy in geodesic calculation, Karney proposed an improved algorithm in 2013. It is based on series expansions of ellipsoidal geodesic formulae and can handle extreme cases that affect Vincenty's method, such as near-antipodal points, while reaching accuracy close to floating-point limits. Karney's algorithm is implemented in GeographicLib, and many modern GIS tools and programming libraries, such as geopy, use Karney's method by default for geodesic distance. Compared with Vincenty's algorithm, Karney's algorithm converges faster and more reliably, making it suitable for surveying and geographic computation with high accuracy requirements.

• Other methods: In a spherical model, the haversine formula is often used to directly calculate the great-circle distance between two points as an approximation of the geodesic. Because Earth is an oblate spheroid, however, spherical methods can have errors of up to about 0.5% according to sourceforge.net. For example, the error can reach several meters over a 1 km distance and several kilometers on long-distance routes. Therefore, when high accuracy is required, such as in surveying or boundary delineation, ellipsoidal geodesic algorithms should be used.

Projected Distance

Projected distance, also called planar distance or Cartesian distance, is the planar distance obtained after converting coordinates into the corresponding projected coordinate system.

Rhumb Line

Definition and characteristics: A rhumb line, also called a loxodrome, is a curve on Earth's surface that maintains a constant angle with all meridians. In other words, if the compass bearing relative to true north remains unchanged during navigation or movement, the path is a rhumb line. On a Mercator projection map, rhumb lines appear as straight lines, a property that made them extremely important in the history of marine navigation. Compared with a geodesic, a rhumb-line route is not the shortest path between two points. However, because its bearing remains constant, it was commonly used in early marine and aviation navigation to simplify steering.

Typical Uses

Rhumb-line routes have an important place in the history of navigation. In ancient navigation, because precise navigation instruments were unavailable, sailors found it easier to maintain a constant compass bearing and therefore often used rhumb-line routes. For example, sixteenth-century nautical charts used the Mercator projection so that routes could be drawn as straight lines.

In practice, constant-bearing routes are often used at low latitudes or for short-distance and coastal navigation, where the difference between a rhumb line and the shortest path is small and the bearing is easy to maintain. However, at high latitudes or on long transoceanic routes, rhumb lines curve substantially and are clearly longer than geodesics. For example, a constant-bearing route from London to New York is about 4% longer than a great-circle route, and a constant-bearing route from New York to Beijing can be about 30% longer than the shortest path. Modern long-distance navigation therefore mainly uses geodesic or great-circle routes, while rhumb lines are more often used for teaching demonstrations or specialized scenarios, such as meteorological drones cruising along fixed bearings.

Even so, rhumb lines still matter in some cartographic contexts. Nautical charts may include rhumb-line grids so captains can quickly determine bearings from constant-bearing lines. Meteorology may also analyze paths with constant wind direction. In GIS analysis, the rhumb-line model is useful when a movement track with a constant bearing needs to be represented.

Calculation Methods and Algorithms

Rhumb-Line Formula in a Spherical Model

In a spherical model, rhumb lines can be derived using spherical trigonometry. The spherical rhumb-line distance formula is:

$$d = R \cdot \sqrt{ \Delta\phi^2 + q^2 \cdot \Delta\lambda^2 }$$

• $R$ is the sphere radius.
• $\Delta\lambda$ is the longitude difference in radians, using the minimum angular range $[-\pi, \pi]$.

---

Initial course angle, or azimuth, is calculated as:

$$\theta = \arctan\left( \frac{ \Delta\lambda }{ \Delta\psi } \right)$$

Rhumb-Line Formula in an Ellipsoidal Model (Approximation)

On an ellipsoidal model, rhumb-line calculation is more complex because meridian spacing changes with latitude and is no longer uniform. Common approaches include:

1. Approximation method:
   • First calculate the ellipsoidal arc length along the meridian between the two points, which is the pure north-south distance.
   • Then estimate the path length by dividing by $\cos\alpha$, where $\alpha$ is the course angle.
2. Projection-inverse method (more accurate):
   • Convert the two longitude/latitude points to ellipsoidal Mercator projection coordinates, requiring integration of $1/\cos\phi'$.
   • Calculate the straight-line distance directly on the projection plane.
   • Convert that distance back to a length on the ellipsoidal surface.

Although this method involves complex integration, it has small errors and acceptable accuracy over short to medium distances and in low to middle latitudes.

Great Ellipse Line

A great ellipse line is the curve formed by intersecting Earth's ellipsoid with a plane that passes through the ellipsoid center and contains the start and end points. In other words, a great ellipse is a central-section ellipse on the ellipsoid. On a sphere, this section is a great circle, so a "great ellipse line" is equivalent to a "great-circle route" in the spherical case. A great ellipse route is generally not a strict geodesic, except in special cases such as the equator or a meridian. However, because Earth's flattening is very small, the length of a great ellipse route is very close to the true geodesic length. When the two points are separated by no more than one quarter of Earth's circumference, about 10,000 km, the difference is only about one part in 500,000. Therefore, a great ellipse route is sometimes treated as a good approximation for long-distance navigation.

Applicable Scenarios

Great ellipse routes mainly played a role historically in long-distance marine and aviation planning. When a voyage crosses a large ocean area or involves high latitudes, a traditional rhumb-line route becomes significantly longer, while a great-circle route is shorter but does not account for Earth's flattening. Navigators therefore used a "great ellipse route" for precise computation to obtain distances more accurate than great-circle distances. For example, early ocean-going ships sometimes used the great ellipse method in charting to correct great-circle route errors and make route length closer to the actual shortest path.

In modern applications, because computers can solve geodesics directly, the practical need for great ellipse routes as an independent method has decreased.

In teaching and research, however, it remains an important concept for understanding the difference between spherical and ellipsoidal routes. Navigation schools may compare rhumb-line, great-circle, and great ellipse routes under different projections. Aviation also considers the role of great ellipses in polar route planning, such as flights crossing polar regions where the route plane approximately passes through Earth's center. Overall, when simplified computation is desired while maintaining high accuracy, a great ellipse route can be a compromise: it is simpler to calculate than a geodesic while having nearly equivalent accuracy, which is why it was once used for manual planning of long-distance routes.

In current GIS applications, however, the "great ellipse" path is usually distinguished only in specific analyses or professional simulations. For ordinary long-distance shortest-path problems, using a geodesic algorithm directly is sufficient. Major GIS software and tools also tend to provide geodesics as the default option to ensure the true shortest-distance result. Therefore, great ellipse lines are mainly theoretical and educational, helping explain subtle differences in route planning caused by Earth's non-spherical shape.

Calculation Principles and Algorithms

Calculating a great ellipse line usually requires solving the intersection between a plane and an ellipsoid. The basic principle is that the geographic coordinates of the two endpoints determine the plane equation passing through the ellipsoid center and the two points. The intersection between that plane and the ellipsoidal surface is then calculated. Mathematically, this yields the parametric equation of a conic section, specifically an ellipse. Direct solution can be complex, so GIS commonly uses an equivalent spherical mapping method to simplify the calculation:

• Auxiliary sphere method: Project or map Earth's ellipsoid to an auxiliary sphere so that the great ellipse containing the two points corresponds to a great circle on the sphere. Common methods include mapping the ellipsoid's parametric latitude, also called the reduced latitude, or geocentric latitude to the sphere, as described on en.wikipedia.org. For example, select a sphere with the same equatorial radius as the ellipsoid, and transform the latitude $\beta$ on the sphere by satisfying $\tan\beta = (1-f)\tan\phi$, where $f$ is the ellipsoid flattening and $\phi$ is the geodetic latitude, as described on sourceforge.net. In this way, the great ellipse on the ellipsoid maps to a great circle on the sphere, as described on en.wikipedia.org. On the auxiliary sphere, spherical great-circle route formulae can be used to calculate distance and azimuth, and then the result can be transformed back to the original ellipsoidal coordinates to obtain the great ellipse route solution. Literature also includes inverse algorithms specifically for great ellipse routes, such as Williams' 1996 great ellipse distance and azimuth formulae for navigation calculations, as described on en.wikipedia.org.

• Accuracy and differences: Because Earth's flattening is small, about 1/298.3, the length difference between a great ellipse line and a true geodesic is extremely small over common distance ranges. For example, on transatlantic routes spanning several thousand kilometers, the difference may be only meters to tens of meters. Therefore, when electronic computation was inconvenient, the great ellipse was considered a long-route method that was simple to calculate and sufficiently accurate, as described on sourceforge.net. It should be emphasized that a great ellipse is not a strict shortest path, especially over ultra-long distances approaching half of Earth, where small differences occur. However, these differences are very small relative to the total distance, generally less than 0.1%. For modern high-accuracy requirements, the optimal solution can be obtained directly with geodesic algorithms, while the great ellipse mainly remains a concept or approximation method.

Distance Calculation in iXGIS

iXGIS performs geodesic calculation with the PROJ library. Geodesics use Charles F. F. Karney's exact ellipsoidal geodesic algorithm from 2013. See the reference documentation for the geod_inverse function — PROJ 9.6.2 documentation. Researchers interested in the specific algorithm can refer to the original paper, Algorithms for geodesics | Journal of Geodesy, published June 26, 2012, volume 87, pages 43-55.

Karney's algorithm has the following advantages over the earlier Vincenty algorithm (1975):

Comparison Item	Vincenty Algorithm	Karney Algorithm
Accuracy	Millimeter level (high accuracy)	Nanometer level (extremely high accuracy)
Stability	Diverges near antipodal points (180 degrees)	Fully stable convergence
Usable latitude range	Nearly global, but unstable near the poles	Global coverage, stable in polar regions
Users	GPS calculators and early GIS	Modern PROJ and GeographicLib

Projected distance is calculated after using the PROJ library to transform coordinates into the projected coordinate system. It is suitable for short-distance calculations.

Rhumb-line calculation uses the PROJ library and is approximated by measuring a straight line after Mercator projection. In pyproj, this is exposed through Geod.rhumb_inverse() and rhumb_direct().

Great ellipse line calculation uses GeographicLib. GeographicLib is a high-precision geographic computation library developed by American mathematician Charles F. F. Karney for geodetic calculations on an ellipsoid. It is widely used in GIS, navigation, remote sensing, cartography, surveying engineering, and related fields, and is one of the most authoritative and accurate tools in modern geodetic measurement. https://grass.osgeo.org/grass-stable/manuals/d.geodesic.html)