Shortest Distance of a Point in $R^3$ to a Cone I'm having a problem how to figure out the shortest distance of a point $\vec{p} = [x_p, y_p, z_p]$ to the surface of a cone given by:


*

*Start vertex $\vec{a} = [x_a, y_a, z_a]$. This is the center of the bottom circle in $\Bbb{R}^3$.

*End vertex $\vec{b} = [x_b, y_b, z_b]$. This is the center if the top circle in $\Bbb{R}^3$.

*The main axis of the cone is then defined as $\vec{b} - \vec{a}$.

*Base radius $r_a$, top radius $r_b$.

*Height $h$ is defined as $|\vec{b} - \vec{a}|$.


i tried to solve the case for a symmetric cone first but im not coming that far
 A: 3d --> 2d
You can use the rotational symmetry of the cone to reduce the problem to $\Bbb{R}^2$: take the plane which contains $p$, $a$, and $b$, i.e., the plane through the point and the axis of the cone.
Now, the point on the cone closest to $p$ lies on this plane, do all you need to do is to find the distance from a given point to the given isosceles trapezoid (which is the intersection of the cone with the plane).
Distance from a polygon
The distance to a polygon (a trapezoid in your case) is the smallest of the distances from its sides.
The computation of the distance between a point $\vec{p}$ and a segment (a side of the
polygon) with ends $\vec{a}$ and $\vec{b}$ depends on the position of
the projection of $ \vec{p} $ on the segment relative to its ends ($(\cdot,\cdot)$
is the scalar product of two vectors, $||\vec{x}||=\sqrt{(\vec{x},\vec{x})}$ is the length of a vector):
If $(\vec{p}-\vec{b},\vec{b}-\vec{a})\ge 0$, then the projection lies outside of the segment
beyond $ \vec{b} $ and the distance is $||\vec{p}-\vec{b}||$.
If $(\vec{p}-\vec{a},\vec{a}-\vec{b})\ge 0$, then the projection lies outside of the segment
beyond $ \vec{a} $ and the distance is $||\vec{p}-\vec{a}||$.
Otherwise the projection lies inside the segment and the distance is
$$||(\vec{p}-\vec{a}) - \frac{\vec{b}-\vec{a}}{||\vec{b}-\vec{a}||^2}(\vec{b}-\vec{a},\vec{p}-\vec{a})||$$ 
A: 2d -> 3d
In the "3d -> 2d" post, the shortest distance is in the plane through $\vec{p}$ and the axis of the cone. The intersection of this plane and the cone is a trapezium and you need to find the 3D coordinates of each of its vertices. In the picture I have labelled these vertices as $\vec{c}$, $\vec{d}$, $\vec{e}$, $\vec{f}$.

These four points are all a perpendicular distance from the axis. If we can find a direction vector perpendicular to the axis in this plane, then we can make a unit vector in this direction and so find the coordinates of the vertices.
The vector $\vec{x}$ in the diagram is perpendicular to the axis. You can find it by projecting $\vec{p}$ onto the axis and subtracting the result. This was done in the "3d -> 2d" post already:
So:
$$
\vec{x} = (\vec{p} - \vec{a}) - \frac{(\vec{p}-\vec{a},\vec{b}-\vec{a})}{(\vec{b}-\vec{a},\vec{b} - \vec{a})}(\vec{b} - \vec{a})
$$
Dividing this vector by its length gives a unit vector, which I will call $\vec{u}$. That is,
$$
\vec{u} = \frac{1}{||\vec{x}||}\vec{x}
$$
Now we have the following:
$$
\vec{c} = \vec{a} + r_a \vec{u}\\
\vec{d} = \vec{b} + r_b \vec{u}\\
\vec{e} = \vec{b} - r_b \vec{u}\\
\vec{d} = \vec{a} - r_a \vec{u}
$$
From the "3d -> 2d" post, you need to find the shortest distance from $\vec{p}$ to each of the edges of the trapezium, and the minimum of all of these is the shortest distance from $\vec{p}$ to the cone.
