Given 2 points A,B,O on sphere of radius $R$. Point O is in middle of AB. E and F are deviation from O by geodesic distance $d$ (angle between EF and AB is $90^o$). Consider 2 circles $C_1,C_2$ on sphere which go through points A,B,E and $A,B,F$, respectively. Given a point X (x,y,z) on sphere, check $X$ belongs to intersection of $C_1$ or $C_2$.
Output: X lies in area defined by $AEBF$ or not? Every points lie in sphere and the distance between 2 points is given by the great circle distance.
My idea is to first find coordinate of 2 points E and F, then find center and radius of 2 circles which go through ABE and ABF. With this information, checking X in AEBF is very easy. I am able to find the great circle distance between 2 points on sphere. But I have no idea on locating the coordinate of E and F and the circle on sphere with given 3 points.
Does anyone have a better algorithm (a trick)? An approximate solution is also appreciated. Thank you in advance !