I'm interested in finding the equation that will tell me if a given geographical coordinate (lat1, lon1) is within an ellipse centered on a another coordinate (lat2, lon2) with a given semi-major axis (a) and a semi-minor axis (b) and a rotation (rot) in radians.

I've found equations for dealing with points inside of an unrotated ellipse, but none that take into account the curve of the Earth, e.g., using the haversine formula, nor any that involve a rotation.

I thought I could puzzle through this, but I've hit the limits of my own geometry... the goal is to turn this into a SQL query, for whatever that is worth.

  • $\begingroup$ stackoverflow.com/questions/7946187/… $\endgroup$ Commented Jul 7, 2020 at 9:50
  • $\begingroup$ If you can get the ellipse equation and project it into XY - the sign of the value inserted in the equation will tell you if it is inside or outside the perimeter of the ellipse. $\endgroup$
    – Moti
    Commented Jul 8, 2020 at 20:30
  • $\begingroup$ Thanks @Aretino — that code is great, except I don't really understand how to deal with latitude/longitude issue. If I treat the lat/lon just naively as points, I get very warped results (not great circle results) as I move towards the poles. $\endgroup$
    – nucleon
    Commented Jul 8, 2020 at 23:41
  • $\begingroup$ Do you know how to compute the distance between two points on the sphere? If so, you could find foci $F_1$, $F_2$ and check if $PF_1+PF_2\lessgtr 2a$. $\endgroup$ Commented Jul 9, 2020 at 12:09

1 Answer 1


If the centre $C$ of the ellipse has longitude $\phi$ and colatitude $\theta$ then its coordinates are $$C=R(\sin\theta\cos\phi,\sin\theta\sin\phi,\cos\theta),$$ where $R=\ $radius of the sphere.

The foci $F_1$, $F_2$ of the ellipse are located on the major axis, at a distance $c=\sqrt{a^2-b^2}$ from the centre. If the major axis is rotated of an angle $\alpha$ (counterclockwise) with respect to the North direction, then the coordinates of the foci are: $$ F_{1,2}=R \pmatrix{\pm\cos \alpha \sin \gamma \cos \phi \cos \theta\mp\sin \alpha \sin \gamma \sin \phi+\cos \gamma \cos \phi \sin \theta \\ \pm\cos \alpha \sin \gamma \sin\phi \cos \theta\pm\sin \alpha \sin \gamma \cos \phi+\cos \gamma \sin \phi \sin \theta \\ \cos \gamma \cos \theta\mp\cos \alpha \sin \gamma \sin \theta}, $$ where $\gamma=c/R$ is the central angle corresponding to the distance between the centre and a focus.

To check if a point $P$ on the sphere is inside the ellipse, compute its distances $d_1$, $d_2$ from the foci and check if $d_1+d_2\le2a$.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .