Equation of pair of lines created by the intersection of 2 double cones? The equations of $2$ double cones having their vertex at the origin $(0,0,0)$ are given by:
$(ax+by+cz)^2=(x^2+y^2+z^2)\cos^2(\theta_1)   \hspace{25pt} (1)$
($\theta_1=$semi-apical angle, and $a,b,c$ are the direction cosine of the axis of the cone)
$(a'x+b'y+c'z)^2=(x^2+y^2+z^2) \cos^2(\theta_2) \hspace{25pt} (2)$
($\theta_2=$ semi-apical angle, and $a',b',c'$ are the direction cosine of the axis of the cone)
Suppose, it's given that the 2 cones intersect. So, it is obvious that their intersection will yield a pair of straight lines (Also at a specific case only a line, when just touching each other).
What will be the equation of the lines (or line)?
 A: Define $r = [x, y, z]^T$ and
$Q_1 = \cos^2(\theta_1)I- v_1 v_1^T $
$Q_2 = \cos^2(\theta_2) I- v_2 v_2^T $
where $v_1 = [a, b, c]^T , v_2 = [a', b', c']^T $
Then the equation of the cones are
$r^T Q_1 r = 0 $
and
$r^T Q_2 r = 0 $
If $r$ satisfies both equations (i.e. $r$ lies on both cones) then
it will satisfy
$r^T Q_\alpha r = 0 $
where
$Q_\alpha = Q_1 + \alpha Q_2 $
The determinant of $Q_\alpha$ is a cubic polynomial in $\alpha$ that has at least one real root.  Choosing this value for $\alpha$, we can diagonalize $Q_\alpha$ into
$Q_\alpha = R D R^T $
where $D$ has one of its diagonal entries equal to zero.  Therefore, $D$ takes the form
$D = \begin{bmatrix} D_{11} && 0 && 0 \\ 0 && D_{22} && 0 \\ 0 && 0 && 0 \end{bmatrix} $
Now we have the following situation
$ r^T R D R^T r = 0 $
Define vector $u = R^T r$, then $ u^T D u = 0 $ so that
$D_{11} u_1^2 + D_{22} u_2^2 = 0 $
If $D_{11}$ and $D_{22}$ have the same sign, then the above equation implies that
$u_1 = u_2 = 0$ while $u_3 = t$, i.e. $u = t [0, 0, 1]^T$
Hence candidate solutions are $r = t R_3 $ where $R_3$ is the third column of matrix $R$.  Now using the equation of the first cone, we get
$ t^2 R_3^T Q_1 R_3 = 0 $
If $R_3^T Q_1 R_3 \ne 0 $ then $t = 0 $ and the origin is the only solution, otherwise $t$ can be any real number, and $r = t R_3$ is the single line of intersection between the two cones.
Next, we assume that $D_{11}$ and $D_{22}$ have different signs, then from
$D_{11} u_1^2 + D_{22} u_2^2 = 0 $
we have
$ u =  \begin{bmatrix} t \\ \pm t \sqrt{ \dfrac{D_{11}}{D_{22}} } \\ s \end{bmatrix}$
where $t$ and $s$ are real.
Vector $u$ is more conveniently written as
$ u = t x_1 + s x_2 $
where
$x_1 = [1 , \pm  \sqrt{ \dfrac{D_{11}}{D_{22}} } , 0 ]^T$
$x_2 = [0, 0, 1]^T$
Now $r = R u = R ( t x_1 + s x_2) = t (R x_1) + s (R x_2 ) = t y_1 + s y_2$
Using $r^T Q_1 r = 0 $ we get
$t^2 y_1^T Q_1 y_1 + 2 t s y_1^T Q_1 y_2 + s^2 y_2^T Q_1 y_2 = 0 $
Dividing through by $s^2$ and defining $ \beta = \dfrac{t}{s} $, then
$ \beta^2 (y_1^T Q_1 y_1) + 2 \beta (y_1^T Q_1 y_2) + (y_2^T Q_1 y_2) = 0 $
which is a quadratic equation in $\beta$, and can have no solutions, implying no lines of intersection, one solution giving a single line of intersection, or two solutions implying two lines of intersection.
Suppose, for example, that we have two solutions $\beta_1 , \beta_2 $ then
for the $i$-th line we have
$ t = \beta_i s $
and
$r = t y_1 + s y_2 = s ( \beta_i y_1 + y_2 ) $
for $i = 1, 2$
And this gives the equation of the lines of intersection as a scalar multiple of the vector $ (\beta_i y_1 + y_2 ), i = 1, 2$
A: Another way to solve this problem is to consider the lines that constitute the curved surface of the first cone, and apply the condition that the angle it makes with the axis of the second cone be $\theta_2$.
A line on the first cone is generated as follows.  Let $d$ be defined as
$d = a_1 + \tan( \theta_1) u_1 $
where $a_1= [a, b, c]$ is the unit vector along the axis of the first cone, and $u_1$ is any unit vector that is orthogonal to $a_1$.
Then the set of all scaled rotations of $d$ about the axis $a_1$ constitutes the surface of the first cone.  Rotations of $d$ are about the axis $a_1$.  Thus, a general line on the surface of the first cone is given by
$ r = t R d $
where $R$ is the rotation matrix having axis $a_1$ and angle of rotation $\phi$, and $t \in \mathbb{R} $
Using the Rodrigues' rotation matrix formula we can express $R$ as
$R = {a_1 a_1}^T + (I - {a_1 a_1}^T ) \cos(\phi) + S_{a_1} \sin(\phi) $
where
$ S_{a_1} = \begin{bmatrix} 0 && - a_{1z} && a_{1y} \\
a_{1z} && 0 && -a_{1x} \\
-a_{1y} && a_{1x} && 0 \end{bmatrix} $
Now we want vector $r$ to lie on the second cone, hence the angle it makes with $a_2=[a',b',c']^T$ has to be equal to $\theta_2$
Therefore, the condition for intersection can be expressed as
$ \cos(\theta_2) = \dfrac{ t R d \cdot a_2 } { \| t R d \| } = \dfrac{ R d \cdot a_2 }{ \| d \| } $
Now vector $d$ is a constant vector , so this becomes
$ \| d \| \cos(\theta_2) = a_2^T R d $
expanding the right hand side using the expression for $R$ given above, this becomes the simple trigonometric equation
$ A \cos(\phi) + B \sin(\phi) = C $
where
$A = a_2^T (I - {a_1 a_1}^T ) d $
$B = a_2^T S_a d $
$C =  \| d \| \cos(\theta_2) -  a_2^T {a_1 a_1}^T d$
And this equation is trivial to solve.  For completeness, I am going to include it solution.
Find the angle $\psi = \text{Atan2}( A, B ) $ , i.e. the angle $\psi $ satisfies $\cos \psi = \dfrac{A}{\sqrt{A^2 + B^2}} $ and $\sin \psi = \dfrac{B}{\sqrt{A^2 + B^2 }} $, and this transforms the trigonometric equation to
$ \cos \psi \cos \phi + \sin \psi \sin \phi = \dfrac{C}{\sqrt{A^2 + B^2}} $
which simplifies to
$ \cos(\phi - \psi ) = \dfrac{ C }{\sqrt{A^2 + B^2}} $
Hence,
$ \phi = \psi \pm \cos^{-1} \left( \dfrac{C}{\sqrt{A^2 + B^2 }} \right) $
Depending on the values of $A,B,C$ we can have no solutions, a single solution, or two solutions.
Once the values of $\phi $ are found, we find $R$, then we find $r = t R d$ which the parametric form of the line with parameter $t$ and direction $Rd$.
As an explicit example, suppose
$a_1 = [\frac{1}{3}, \frac{2}{3}, \frac{2}{3}]^T $
and
$a_2 = [\frac{1}{9}, - \frac{-4}{9}, \frac{8}{9} ]$
And $theta_1 = \dfrac{\pi}{3} $ and $\theta_2 = \dfrac{\pi}{4} $
We have to solve the trigonometric equation
$A \cos \phi + B \sin \phi = C$
where
$A = a_2^T (I - {a_1 a_1}^T ) d $
$B = a_2^T S_a d $
$C =  \| d \| \cos(\theta_2) -  a_2^T {a_1 a_1}^T d$
and
$ d = a_1 + \tan(\theta_1) u_1 $
and $u_1$ is any unit vector perpendicular to $a_1$
So let's choose $u_1 = [2, 0, -1]/\sqrt{5} $, then
$d = (-1.21586, 0.66666, 1.4412633 ) $
And applying the formula given, we get two solutions for $\phi$,
$ \phi_1 = 0.4016017696 , \phi_2 = 2.09648977 $
And this results in two solutions
$X_1 = t (-0.44537, 0.081013 , 0.891673 ) $
and
$X_2 = t ( 0.778705 , -0.22501 , 0.585654 )$
The above vectors have been normalized.
