Length of loxodrome (rhumb line) between two points on sphere

I have a sphere with two points on its surface: $A=[x_1, y_1, z_1]$ and $B=[x_2, y_2, z_2]$.

I want to find a loxodrome (rhumb line, spherical spiral) with a shortest length that connects these two points and obtain its length. I want the point $A$ to be the starting point of the curve (so something like pole, but its coordinate is not located at pole, but somewhere on the sphere in general).

Something like this: for two different sets of $A_{red}$, $B_{red}$ and $A_{blue}$, $B_{blue}$ I want length of "red" and "blue" loxodrome curve • You want to manage each point as it was one of the poles, don't you? And, are the starting and ending point antipodals? – Rafa Budría Jun 2 '17 at 13:46

A rhumb line is defined by two predetermined poles of the sphere. If the poles of the sphere are where the gray lines converge, you might get a loxodrome like the red curve, but never one between $A_{blue}$ and $B_{blue}$ that looks at all like the blue curve. There would be infinitely many loxodromes between $A_{blue}$ and $B_{blue},$ but none with those tight spirals near the endpoints.

So I'll consider the problem this way: we put arbitrary points $A$ and $B$ on a sphere and then place a pole $P$ at an arbitrary location. The location of the other pole $P'$ is then determined. Then we construct a loxodrome between $A$ and $B$ using $P$ and $P'$ as poles.

A loxodrome has a constant angle $\alpha$ relative to the direction to the pole. Let the latitude and longitude of $A$ and $B$ be $(\lambda_A, \theta_A)$ and $(\lambda_B, \theta_B),$ respectively, all measured in radians (not degrees). Then as already explained in answers to other questions (including Length of loxodrome and How to use a Rhumb Line?), the length of the loxodrome is $\lvert R(\lambda_A - \lambda_B) \sec \alpha\rvert,$ where $R$ is the radius of the sphere.

Since $R,$ $\lambda_A,$ and $\lambda_B$ are all determined by the setup of the problem, the only choice we have is which angle $\alpha$ to use. Except in the case where $\lambda_A = \lambda_B,$ there are infinitely many choices of $\alpha$ that will work; see When is a loxodromic curve unique between two points? for details. But any choice of $\alpha$ must satisfy the equation $$\theta_A - \theta_B + 2k\pi = \ln \left( \frac{\tan(\lambda_B/2 + \pi/4)} {\tan(\lambda_A/2 + \pi/4)}\right) \tan\alpha, \tag1$$

adapted from this answer with the added term $2k\pi$ to account for the fact that we are allowed to go either "westward" or "eastward" and "wrap around" the circle of longitudes as many times as we like. Note that this formula assumes that latitude is measured in the geographic manner, that is, $\lambda = \pm\frac\pi2$ at the poles $P$ and $P'.$

In $(1),$ everything is predetermined except $k$ and $\alpha,$ so we just need to choose $k$ so as to minimize $\lvert\sec \alpha\rvert.$ Since $\sec^2\alpha=1+\tan^2\alpha,$ this means we want to minimize $\lvert\tan\alpha\rvert.$ And since $\tan\alpha$ is just some fixed multiple of $\theta_A - \theta_B + 2k\pi,$ that means we want to minimize $\lvert\theta_A - \theta_B + 2k\pi\rvert.$ So we just choose $k$ according to that criterion. Plot points at angles $\theta_A$ and $\theta_B$ on a unit circle, choose the shorter of the two arcs between those points, and the length of that arc is $\lvert\theta_A - \theta_B + 2k\pi\rvert.$ Plug that into $(1)$ and solve for $\tan\alpha,$ from which you can get $\sec\alpha,$ from which you can get the length of the loxodrome.

A little thought should make it clear that the shortest loxodrome will never have a tight spiral at the end, even if the endpoint $A$ is very near to a pole. And if we put $A$ exactly at a pole, the loxodrome from $B$ to $A$ is a great circle (we head due north to get to the north pole).