Trigonometry, how to order law of sine/cosine operations to get a minimally convoluted term Given is this seemingly trivial problem:

Two elevations ("mountains") on a sphere with radius r, height normals H1, H2 with "summit points" G1 and G2, respectively. Their distance D is given as the length of the great circle arc length between their base points.
How tall can a third elevation H (which is located on the arc between the basepoints of H1 and H2) at an arc length distance d from H1 become, before it obstructs G2 as seen from G1 or vice-versa?

Assumptions made:

*

*Simplified the problem to two dimensions

*Heights are surface normals so they originate from, and intersect at, the circle center

*0 < d < D

*No obstruction between G1 and G2 by the surface arc alone (no negative values for H)

Drawing:

This is how far I got before I realized that this approach must be somehow flawed:

*

*The central angle between H1 and H2 in radians is

φ = D / r


*The central angle between H1 and H in radians is

β = d / r


*The radius is just a constant, thus we assume

h1 = H1 + r
h2 = H2 + r


*Distance G between summit points G1 and G2 with law of cosines:

G² = h1² + h2² - 2 * h1 * h2 * cos(φ)


*Angle α as difference to 180° (π rad) of the angle between line g and H2 with law of sine

α = π - φ - arcsin((h1 * sin(φ)) / G)


*Angle γ similarly from the triangle h1_h_g

γ = π - α - β


*Missing length h (Hx+r) with law of sines:

h = (h1 * sin(α)) / sin(γ)


*Finally, subtract the constant radius

H(d) = h(d) – r
When I attempted to find a formula for the unknown height H as a function H = f(H1, H2, D, d), somewhere between steps 7 and 8, I must have painted myself in a corner. In particular 4, the G² term looks like it would fit without taking the square root, but does not. The term that I get from resubstituting the parts is utterly convoluted. There has to be an easier way to simplify the law-of-cosine term. Or I'm on the wrong track altogether?
 A: There is a seemingly simpler approach using mainly analysis instead of geometry, by using polar coordinates $(\rho,\theta)$. Here is how.
Indeed, the polar equation

*

*of the circle is $\rho=r$.


*of the line is $\rho=\rho_0\frac{1}{\cos(\theta-\theta_0)}$ (see explanations here)
where $(\rho_0,\theta_0)$ are the polar coordinates of the point of the line situated at the shortest distance from the origin.
Therefore, the heights of the different mountains "above sea level" have a common expression :
$$H(\theta)=\rho_0\frac{1}{\cos(\theta-\theta_0)}-r \tag{1}$$
It remains to turn the different constraints into equations :
Taking the axis of mountain n°1 as the reference axis for angles and turning clockwise, (1) gives rise to the two equations:
$$H(0)=\rho_0\frac{1}{\cos(0-\theta_0)}-r=H_1 \tag{2}$$
$$H(\varphi)=\rho_0\frac{1}{\cos(\varphi-\theta_0)}-r=H_2 \tag{3}$$
The system (2)+(3) can be written:
$$\begin{cases}\rho_0&=&\cos(\theta_0)(r+H_1) \\ \rho_0&=&\cos(\varphi-\theta_0)(r+H_2)\end{cases}$$
giving the equation :
$$\cos(\theta_0)(r+H_1)=\cos(\varphi-\theta_0)(r+H_2)\tag{4}$$
from which we can deduce unknown $\theta_0$ (in an explicit form : see formula (6) in Appendix below). Plugging the value of $\theta_0$ into (2) provides the value of
$$\rho_0=(r+H_1) \cos(\theta_0)$$.
Now that we know $\rho_0$ and $\theta_0$, we are able to express the variable altitude "above sea level" of intermediate mountain by the very same type of equation as (2) or (3), with generic angle $\theta$ replaced by $\alpha$ :
$$H(\alpha)=R=\rho_0\frac{1}{\cos(\alpha-\theta_0)}-r$$
which is the height of the "just non-occulting mountain" for angle $\alpha$.
Here is a graphical representation (with GeoGebra) of function $\alpha \mapsto H(\alpha)$. I have taken values sticking to those in your figure : $r=1, \varphi=\pi/3, R_1=0.8, R_2=0.4$. See particular points (little blue disks) corresponding to extreme positions, with a minimum occuring (grey disk) with coordinates $(\theta_0,\rho_0)$ between these two extreme positions, as awaited.
Caution about the names of variables : with Geogebra, we are not free to give them the name we want ; it is why the final equation (corresponding to equation (1)) of variable $\theta$ uses variable $x$ instead of $\theta$. Variables $t0, r0$ are for $\theta_0,\rho_0$.
Appendix : solution of equation (4):
Expanding (4) into :
$$\cos(\theta_0)(r+H_1)=\cos(\varphi)\cos(\theta_0)(r+H_2)+\sin(\varphi)\sin(\theta_0)(r+H_2)\tag{5}$$
which is equivalent to :
$$\frac{\sin(\theta_0)}{\cos(\theta_0)}=\frac{(r+H_1)-\cos(\varphi)(r+H_2)}{\sin(\varphi)(r+H_2)}$$
we conclude that,
$$\theta_0=\tan^{-1}\left(\frac{(r+H_1)-\cos(\varphi)(r+H_2)}{\sin(\varphi)(r+H_2)}\right)\tag{6}$$
(this formula is valid for a certain range of values : with inverse trigonometric functions, one must always be cautious !).

A: The area of the big triangle equal to the sum of areas of the two smaller triangles
$$\sqrt{s(s-h_1)(s-h_2)(s-G)} = \frac{1}{2}h_1 h_x \sin \beta + \frac{1}{2} h_2 h_x \sin(\varphi - \beta)$$
Where $h_x = r + H_x$ and $s = \frac{h_1+h_2+G}{2}$
