Given three rays and a right triangle, draw the pyramid having those rays as edges and that triangle as base I am looking for suggestions for solving the following geometrical problem algebraically.
I have three rays in the 3D space emanating from an origin $O$, say $\alpha$, $\beta$, $\gamma$.
I wanted to find a plane $\pi$ intersecting these three lines in the points $A$, $B$, $C$ respectively, so that the triangle $ABC$ is right, say in $B$, and that its sides have known lengths $\overline{AB}$ and $\overline{BC}$. Obviously, $\overline{AC}^2$ = $\overline{AB}^2$ + $\overline{BC}^2$.
Basically, given three rays out of an origin and a right triangle, the plane $\pi$ draws a pyramid having those rays as the edges and that right triangle as the base. It does not matter whether the pyramid is acute or obtuse.
I also understand that the problem may have no solution in particular circumstances. When a solution exists, though, I would like to determine the position of the intersection points $A$, $B$ and $C$ between thosee rays $\alpha, \beta, \gamma$ and that plane $\pi$.

I cast the problem using these symbols: the pyramid edges are the vectors $\vec{a}=OA$, $\vec{b}=OB$, $\vec{c}=OC$; and their unknown lengths are $a = |\vec{a}|, b = |\vec{b}|, c = |\vec{c}|$.
I use the law of the cosines to express the fact that the faces of the pyramid are generic triangles:
$ a^2 - 2 \cos \hat{\alpha\beta}\, ab  + b^2 = \overline{AB}^2$
$ b^2 - 2 \cos \hat{\beta\gamma}\, bc  + c^2 = \overline{BC}^2$
$ c^2 - 2 \cos \hat{\gamma\alpha}\, ca  + a^2 = \overline{CA}^2$
and the Pythagorean theorem to state that the base of the pyramid is a right triangle:
$\overline{AB}^2 + \overline{BC}^2 = \overline{AC}^2 $
I have three nonlinear equations in the three variables $a, b, c$ and one condition. I lack a strategy to solve this system and obtain closed-form formulas for the solution.

My questions are

*

*As a sanity check, does the formulation express the problem well in the first place?


*How can I then solve for  $a,b,c$ in the system above?
 A: The nonlinear system of equations can be solved successfully and easily with the Newton-Raphson multivariate method.  The vector function is
$ f(a,b,c) = \begin{bmatrix} a^2 + b^2 - 2 a b \cos t_1 - X^2 \\ a^2 + c^2 - 2 a c \cos t_2 - Y^2 \\ b^2 + c^2 - 2 b c \cos t_3 - (X^2 + Y^2) \end{bmatrix}$
Where $t_1$ is the angle between $v_1$ and $v_2$ and $t_2$ is the angle between $v_1$ and $v_3$ and $t_3$ is the angle between $v_2$ and $v_3$.  $X$ is the length of the first leg of the right triangle and $Y$ is the length of the second leg.
Newton-Raphson method uses the Jacobian matrix of the vector function $f$.  The ${ij}$-th entry of this matrix is the partial derivative of $i$-th function with respect to the $j$-th variable.  Thus the Jacobian is given by
$J(a,b,c) = \begin{bmatrix} 2 a - 2 b \cos t_1 && 2 b - 2 a \cos t_1 && 0 \\
2a - 2 c \cos t_2 && 0 && 2 c - 2 a \cos t_2 \\
0 && 2 b - 2 c \cos t_3 && 2 c - 2 b \cos t_3 \end{bmatrix}$
Start with a good guess of the solution $x_0 = (a,b,c)$, then update your guess with
the Newton-Raphson recursive update formula
$ x_{k+1} = x_k - J^{-1} f(x_k) $ ,   $k = 0, 1, 2, ... $
This method is very robust and will lead to a solution if it exists within $4-5$ iterations for this small number of variables (3 variables in this problem).
The only catch is you need to have a routine to find the inverse of a matrix (because you need to calculate $J^{-1}$).
A: Let $\vec{OA}=\vec a=\lambda u$, $\vec{OB}=\vec b=\vec v$ and $\vec{OC}=\vec c=\mu\vec w$ representing the three rays.  Any other solution may be achieved by scaling.
Please note that there's not always a solution, e.g., if $\vec u$, $\vec v$, and $\vec w$ are pairwise orthogonal.  And you can't choose the lengths of $AB$ and $BC$ independently.
For a right angle at $B$ we must have
$$\langle\vec{BA},\vec{BC}\rangle=
\langle\lambda\vec u-v,\mu\vec w-u\rangle=0.$$
Solving for $\mu$ gives
$$
\mu=\frac{\lambda\langle u,v\rangle-\langle\vec v,\vec v\rangle}{\lambda\langle u,v\rangle-\langle\vec v,\vec w\rangle}.
$$
Now if for example
$$
\vec u=\begin{pmatrix}1\\1\\0\end{pmatrix},
\vec v=\begin{pmatrix}1\\0\\1\end{pmatrix},
\vec w=\begin{pmatrix}0\\1\\1\end{pmatrix}
$$
and $\lambda=3$ you'll get $\mu=1/2$ and
$$
\vec{BA}=\begin{pmatrix}3\\2\\-1\end{pmatrix},
\vec{BC}=\begin{pmatrix}-1\\1/2\\-1/2\end{pmatrix}.
$$
If the length of $BA$ is $1$, let $\vec u$, $\vec v$, and $\vec w$ be unit vectors. As $\vec{BA}=\lambda\vec u -\vec v$ we get
$$
1=\|\vec{BA}\|^2=\|\lambda\vec u-\vec v\|
=\lambda^2-2\lambda\langle\vec u,\vec v\rangle+1$$
and from here
$$\lambda=\frac{\langle\vec u,\vec v\rangle}{2}.$$
Now compute $\mu$ and finally $\vec{OC}$.
If $AB$ has a length different from $1$, scale accordingly.
