Rotating a vector around an axis so that it falls onto the specified plane I'm trying to refactor/solve the following equation for theta if $ \vec{v} , \vec{r} $ and $ \vec{n} $ are known vectors:
$ \left(\vec{v} \cdot \cos\left(\theta\right) + \left(\vec{r} \times \vec{v}\right) \cdot \sin\left(\theta\right) + \vec{r} \cdot \left(\vec{r} \cdot \vec{v}\right) \cdot \left(1 - \cos\left(\theta\right)\right)\right) \cdot \vec{n} = 0 $
If this looks familiar, it's probably because it is - it's a combination of the Rodrigues equation for rotating a vector (in this case, $ \vec{v} \,$) around an axis $ \left(\vec{r} \right)$ and a simple dot operation to determine the distance of the resulting vector to a given plane defined by a normal $ \left(\vec{n}\right) $.
In essence, I'm trying to solve for $ \theta $ such that the vector $ \vec{v} $ will be rotated onto plane $ \vec{n} $ if possible. Near as I can tell, there should be either $ 0, 1 $ or $ 2 $ solutions to this problem - making me think that it might be possible to refactor this into something that can be solved by a quadratic equation.
Alternatively (and I have no idea how this would work) - I'm guessing it might also be possible to somehow "project" $ \vec{v} $ around the axis $ \vec{r} $ so that it falls on the plane with a normal defined by $ \vec{n} $ and then calculate the angle between the original vector and the projected vector $ \left(\vec{s}\right) $, but I'm not really sure how you would do that.
How can I solve this problem?
 A: The tip of the rotated vector $v'$ as an image of rotation of vector $v$ about an axis $a$ describes a circle in $3D$.  Assuming the plane in question passes through the origin, the we need this tip of $v'$ to lie in the plane.
Decompose $v$ along the axis of rotation and perpendicular to it as $v_1$ and $v_2$ which are defined as follows
$ v_1 = (  v \cdot a )a , v_2 = v - v_1 $
where $a$ is assumed to be a unit vector.
Define $v_3 = a \times v_2 $
Now the rotated vector as a function of the rotation angle $\theta$ is given by
$ v' = v_1 + v_2 \cos \theta + v_3 \sin \theta $
which we want to satisfy
$ n \cdot v' = 0 $
This means that we want to solve the following equation for $\theta$:
$ (v_2 \cdot n) \cos \theta + (v_3 \cdot n) \sin \theta = - (v_1 \cdot n) $
And this equation can produce 0, 1, or 2 solutions.
A: Starting from your equation $$\left(\vec{v} \cdot \cos\left(\theta\right) + \left(\vec{r} \times \vec{v}\right) \cdot \sin\left(\theta\right) + \vec{r} \cdot \left(\vec{r} \cdot \vec{v}\right) \cdot \left(1 - \cos\left(\theta\right)\right)\right) \cdot \vec{n} = 0$$ gives $$a \cos \theta - b \sin \theta = c,$$ where $a=\left(\vec{n} \cdot \vec{r} \right) \left(\vec{r} \cdot \vec{v}\right) - \vec{n} \cdot \vec{v}$, $b=\vec{n} \cdot \left(\vec{r} \times \vec{v}\right)$ and $c=\left(\vec{n} \cdot \vec{r} \right) \left(\vec{r} \cdot \vec{v}\right)$. Let $M=\sqrt{a^2+b^2}$ and $\alpha = \arctan \dfrac{b}{a}$; then $$M \cos \left(\theta + \alpha\right) = c,$$ which will give two values of $\theta$ if $|c| < |M|$, one if $|c| = |M|$ and none if $|c| > |M|$.
A: In the case where the cone intersects the plane orthogonal to $\vec{n}$ (that we will call the horizontal plane) along two of its generatrices, let $2\varphi$ be the angle between these 2 generatrices.
Let
$$\alpha=angle(r,v) \ \text{and} \ \ \beta=angle(r,n)$$
and $$a:=\frac{1}{\tan(\alpha)\tan(\beta)}$$

*

*If $a \in [-1,+1]$, then angle $\varphi$ is such that :

$$\sin \varphi = a =\frac{1}{\tan(\alpha)\tan(\beta)}\tag{1}$$
(using a certain convention for angles that will be clear from the explanation below).

*

*Otherwise, there are no solutions.

Here is why.
I use for the proof unitarized vectors $v,r,n$, without loss of generality.
Let us consider the orthonormal basis of the plane orthogonal to $r$ :
$$V_1=\tfrac{1}{k}(r \times n), \ \ V_2=\tfrac{1}{k}(r \times n) \times r \tag{2}$$
where $k=\|r \times n \|= \sin \beta$ is a normalization factor.
Considering now the 3D orthonormal basis $V_1,V_2,r$, we have the following decomposition of the rotated vector $v_{\varphi}$ of $v$ around $n$:
$$v_{\varphi}=\sqrt{1-(v \cdot r)^2}\left(\cos \varphi \ V_1 + \sin \varphi \ V_2\right) \ + \  (v \cdot r) \ r$$
(sanity check : verify that $v_{\varphi}$ has a unit norm)
Now, let us express that we must have the following zero dot product (in the same way as you have expressed the problem):
$v_{\varphi} \cdot n = 0 \tag{3}$
It suffices now to replace in (3) $V_1$ and $V_2$ by their expressions from (2), and take into account the fact that $r \cdot n = \cos \beta$ and $r \cdot v = \sin \beta$ gives relationship (1).

Edit: it remains to get angle $\theta$ from angle $\varphi$ or more precisely from the knowledge of $V_1$ and $V_2$. The idea is to operate in the plane defined by points $V_1,V_2,v$ ("tips of associated vectors"), whose center is the projection of the origin. In this way the rotation angle bringing $v$ onto $V_1$ or to $V_2$ for example will be easily obtained.

Fig. 1: $\textit{All vectors are with unit length. If one projects $V_1,V_2,v$ on the plane}$ $\textit{ defined by the "tips" of $V_1,V_2,v$, one has a circle where one can visualize}$ $\textit{ in true size angles $\theta=\angle(v,V_1)$ and $\theta=\angle(v,V_2)$ and act on them}$.
A: You have the plane containing $\bf r$ and $\bf v$, whose unitary normal is $${\bf w} = \frac{{{\bf v} \times {\bf r}}}{{\left\| {{\bf v} \times {\bf r}} \right\|}}
$$.
You just have to rotate that plane onto the target one, i.e. $\bf  w $ onto $ \bf n$, or onto $\pm \bf n$
depending on whether and how you consider the orientations.
Then it is a simple matter to find the rotation angle through $ \bf u = \bf w \times \bf n$ ,
first having verified that it is parallel to $\bf r$, because otherwise it means that the rotation axis is not
the intersection of the two plane.
