Cone Section: Intersection between a plane ax + by + cz = d and two cones of height $h$, raidus $r$ whose apexes lie at the centre, along the $z$ axis A double cone can be described by circles that increase with $|z|$ and have radius $R$ at $h = |z_R|$. It holds:
\begin{align}
    \frac{R^2}{h^2}\,z^2 = x^2 + y^2
\end{align}
The plane
\begin{align}
    ax + by + cz = d
\end{align}
should now intersect this cone. To do this, we move the plane to $z$ and insert this into the double cone equation.
\begin{align}
    &z = \frac{d - ax - by}{c} \\
&\Longrightarrow \hspace{10pt} \underbrace{\frac{R^2}{c^2h^2}}_{\color{red}{=\,\kappa^2}}\,(d-ax-by)^2 = x^2 + y^2 \\
&\Longrightarrow \hspace{10pt} \kappa^2\,(d^2+a^2x^2+b^2y^2-2adx-2bdy + 2abxy) = x^2 + y^2 \\
&\Longrightarrow \hspace{10pt} \left(\kappa^2a^2-1\right)x^2+\left(\kappa^2b^2-1\right)y^2 + \left(2\kappa^2 ab\right)xy + \left(-2\kappa^2ad\right)x + \left(-2bd\right)y + \kappa^2d^2 = 0
\end{align}
Unfortunately, this is not yet the right solution. If I take, for example, the plane
\begin{align}
    -h \cdot x + 0 \cdot y + 2R \cdot z = Rh
\end{align}
then I get:
\begin{align} 
     &\kappa^2 = \frac{1}{4h^2} \\
     &\Longrightarrow \hspace{10pt} \left(\frac{1}{4h^2}h^2-1\right)x^2-y^2 + \left(-2\,\frac{1}{4h^2}\,(-h)\,Rh\right)x + \frac{1}{4h^2} R^2h^2 = 0 \\
     &\Longrightarrow \hspace{10pt} -\frac{3}{4}x^2-y^2 + \frac{R}{2}\,x + \frac{R^2}{4} = 0 \\
     &\Longrightarrow \hspace{10pt} \frac{3}{R^2}x^2 + \frac{4}{R^2}y^2 - \frac{2}{R}\,x = 1 \\
\end{align}
And I also know where the problem is: I am supposed to look at the function in the sectional plane. So somehow I have to transform my coordinates from the original coordinate system into a coordinate system of the plane. Unfortunately, I don't know how. Maybe the task would even be easier if I had transformed my system right at the beginning. This way I would only have to consider my cone at $z=0$. My approach so far has been as follows:
My origin should be the point $(0,0,\frac{d}{c})$. My new $z$-axis is defined by the normal of the plane, so $\vec{z}_{new} = \frac{\vec{n}}{|\vec{n}|}$ with $\vec{n} = (a,b,c)$. My new $x$-axis shall continue to be perpendicular to the $y$-axis, so $\vec{x}_{new} = (\xi_1,0,\xi_2)$. My new $y$-axis is to be formed by the cross product of these two. Unfortunately, that's as far as I've got so far.
 A: First, you need to express the unit normal to the plane in spherical coordinates:
$n = \dfrac{1}{\sqrt{a^2 + b^2+c^2}} (a, b, c) = (\sin \theta \cos \phi, \sin \theta \sin \phi, \cos \theta) $
Comparing the expressions, we have
$ \theta = \cos^{-1} \dfrac{c}{\sqrt{a^2 + b^2+c^2} }$
$ \phi = \text{ATAN2}(a,b) $
It follows that the steepest descent direction that lies in the plane is
$ u_1 = (\cos \theta \cos \phi, \cos \theta \sin \phi, - \sin \theta ) $
We can take $u_1$ as the $x$ axis in the plane, while the $y$ axis is given by
$ u_2 = (-\sin \phi, \cos \phi , 0 ) $
Using the $u_1, u_2$ axes, any point on the plane can be expressed as
$ r = (x, y, z) = r_0 + u_1 x_1 + u_2 x_2 $
Where $r_0$ is any point on the plane, for example $(0, 0, d / c )$
Written compactly,
$r = r_0 + V u \hspace{48pt} (1)$
with $V = [u_1, u_2] $ and $ u = [x_1, x_2]^T $
Now the equation of the cone is (in terms of $r$)
$ r^T Q r = 0 \hspace{48pt} (2)$
where
$Q = \begin{bmatrix} 1 && 0 && 0 \\ 0 && 1 && 0 \\ 0 && 0 && - \tan^2 \theta_c \end{bmatrix} $
Where $\theta_c$ is the semi-vertical angle of the cone, i.e. $\tan \theta_c = \dfrac{R}{h} $
Substitute $(1)$ into $(2)$
$ (r_0 + V u)^T Q (r_0 + V u) = 0 $
Expand
$ r_0^T Q r_0 + 2 u^T V^T Q r_0 + u^T V^T Q V u = 0 $
Let's find the matrix V^T Q ,
$V^T Q = \begin{bmatrix} \cos \theta \cos \phi && \cos \theta \sin \phi && \tan^2 \theta_c \sin \theta \\ - \sin \phi && \cos \phi && 0 \end{bmatrix} $
Therefore,
$V^T Q V = \begin{bmatrix} \cos^2 \theta - \tan^2 \theta_c \sin^2 \theta && 0 \\ 0 && 1 \end{bmatrix}$
If $ \cos^2 \theta - \tan^2 \theta_c \sin^2 \theta = 0 $, then we have a parabola.
If $ \cos^2 \theta - \tan^2 \theta_c \sin^2 \theta \gt 0 $ then $V^T Q V$ is invertible, and the conic section is an ellipse.
If $ \cos^2 \theta - \tan^2 \theta_c \sin^2 \theta \lt 0 $ then $V^T Q V$ is invertible, and the conic section is an hyperbola.
Let's do the second case.  We can find the center of this ellipse in $u_1u_2$ plane as follows:
$u_0 = - (V^T Q V)^{-1} V^T Q r_0 $
Explicitly evaluating $u_0$, we find
$u_0 = \begin{bmatrix} \dfrac{(-d/c) (\tan^2 \theta_c \sin \theta)}{\cos^2 \theta - \tan^2 \theta_c \sin^2 \theta} \\ 0 \end{bmatrix} $
Then by completing the square, we can write,
$(u - u_0)^T V^T Q V (u - u0) = u_0^T V^T Q V u_0 - r_0^T Q r_0 $
Now, $u_0^T V^T Q V u_0 = \dfrac{(d/c)^2 \tan^4 \theta_c \sin^2 \theta}{\cos^2 \theta - \tan^2 \theta_c \sin^2 \theta}$
and $r_0^T Q r_0 = -(d/c)^2 \tan^2 \theta_c $
Hence,
$u_0^T V^T Q V u_0 - r_0^T Q r_0 = \dfrac{(d/c)^2}{\cos^2 \theta - \tan^2 \theta_c \sin^2 \theta} ( \tan^4 \theta_c \sin^2 \theta + tan^2 \theta_c \cos^2 \theta - \tan^4 \theta_c \sin^2 \theta) $
And this simplifies to,
$ \text{Right Hand Side} = \dfrac{(d/c)^2}{\cos^2 \theta - \tan^2 \theta_c \sin^2 \theta} \cos^2 \theta \tan^2 \theta_c $
Therefore, the semi-major axis of the ellipse is along $u_1$ and given by,
$ a = (d/c) \dfrac{\cos \theta \tan \theta_c }{\cos^2 \theta - \tan^2 \theta_c \sin^2 \theta }$
And the semi-minor axis of the ellipse is along $u_2$ and given by,
$ b = (d/c) \dfrac{\cos \theta \tan \theta_c }{\sqrt{\cos^2 \theta - \tan^2 \theta_c \sin^2 \theta } } $
