3D rotation orientation My problem involves finding a formula for this:
You're given a line that passes through the origin in 3D space (more specifically, a point on the line). You are then given a point and an angle. The task is to find where the point ends up after a counterclockwise rotation around the line by the angle.
I managed to do this by reformulating the problem as an intersection of three spheres, but I got two results: one clockwise and one counterclockwise.
At this point I have no idea how to figure out which is which, because "counterclockwise" isn't a sufficient definition. You must know which way the line is oriented.
Is there a way to choose from the two destination points such that rotating objects (such as a cube, given its vertices) aren't deformed (because the points rotate in different directions)?
 A: Let's say the line passes through origin $(0,0,0)$ and point $\vec{\ell} =(x_\ell , y_\ell, z_\ell)$, the point to be rotated is $(x, y, z)$.
The line represents our rotation axis. Calculate the unit rotation axis $\hat{a} = ( x_a , y_a , z_a )$:
$$\bbox{ \hat{a} = \frac{\vec{\ell}}{\left\lVert\vec{\ell}\right\rVert} = \left( \frac{x_\ell}{\sqrt{x_\ell^2 + y_\ell^2 + z_\ell^2}} , \frac{y_\ell}{\sqrt{x_\ell^2 + y_\ell^2 + z_\ell^2}} , \frac{z_\ell}{\sqrt{x_\ell^2 + y_\ell^2 + z_\ell^2}} \right ) }$$
To rotate point $\vec{p}$ by angle $\theta$ around unit axis vector $\hat{a}$, we can use Rodrigues' rotation formula:
$$\bbox[#ffffef, 1em]{ \vec{p}^\prime = \vec{p} \cos\theta + (\hat{a} \times \vec{p}) \sin\theta + \hat{a} ( \hat{a} \cdot \vec{p} ) (1 - \cos \theta) }$$
where $\vec{p}^\prime$ is the location of the point after the rotation.
If the $\hat{a}(\hat{a}\cdot\vec{p})(1 - \cos\theta)$ throws you off, note that the first term, $\hat{a}$, is a vector, but both the second and third terms are reals.
In case you have forgotten, for $\vec{a} = ( x_a , y_a , z_a )$ and $\vec{b} = ( x_b , y_b , z_b )$, and $c$ a real,
$$\begin{array}{l|l}
\; & c \vec{a} = ( c x_a , c y_a , c z_a ) \\
\hline
\; & \vec{a} + \vec{b} = ( x_a + x_b , y_a + y_b , z_a + z_b ) \\
\hline
\text{Dot product} & \vec{a} \cdot \vec{b} = x_a x_b + y_a y_b + z_a z_b \\
\hline
\text{Cross product} & \vec{a} \times \vec{b} = ( y_a z_b - z_a y_b , z_a x_b - x_a z_b , x_a y_b - y_a x_b ) \\
\hline
\text{Euclidean length or $\ell^2$-norm} & \left\lVert \vec{a} \right\rVert = \sqrt{ \vec{a} \cdot \vec{a} } = \sqrt{ x_a^2 + y_a^2 + z_a^2 } \\
\end{array}$$
A: Let $p$ be the first point lying on the line passing through the origin. Since the line passes through the origin $0$ also lies on it. We can form the unit vector $\frac{p-0}{||p-0||_2} = \hat{p}$, this will be our rotation axis. Let the point we want to rotate around the axis be $v$. After that find the axis-angle rotation matrix $M(\hat{p},\theta)$ (https://en.wikipedia.org/wiki/Rotation_matrix#Axis_and_angle) and the rotate point is $v' = Mv$.
A: You hit the nail on the head!

At this point I have no idea how to figure out which is which, because "counterclockwise" isn't a sufficient definition. You must know which way the line is oriented.

Unless the problem is specified further, there is not really a solution. You could always try to find it experimentally, by simply choosing a scheme to derive the direction for each line (rotational axis).
Some possible schemes are:


*

*Assume that each line is oriented towards the origin.

*Assume that each line is oriented away from the origin.

*Assume that each line is oriented towards positive $x$. (If orthogonal to the $x$-axis, try $y$, then $z$.)


(Since you said that you have lines represented by a point, I am assuming that you will never encounter a point at the origin, since this doesn't specify a line.)
Regardless of the choice made, objects rotated around a particular line won't be deformed, because the same assumption is used in rotating each vertex. This solves the problem, except for that mention of "counterclockwise".
Now, if all of those options work for you, great! But most likely you will find that only one of options 1 and 2 lead to a satisfactory result. Besides having a solution, you will then also know the missing information in the original problem - the meaning of "counterclockwise" in this context.
A: 
Is there a way to choose from the two destination points such that
  rotating objects (such as a cube, given its vertices) aren't deformed
  (because the points rotate in different directions)?

The answer depends; does the choice of "destination points" have to be continuous on the space of lines and angles? Or does it have to be continuous for each line? 
In the latter case, the answer is trivially yes: just pick an orientation. But in the first case I think the answer is no, because a continuous choice of orientation on the space of lines $\mathbb{P^2(R)}$ amounts to a global section of the covering map $\pi: S^2\to\mathbb{P^2(R)}$, which happens not to exist. 
There is also the fact that $\mathbb{P^2(R)}$ is non-orientable, but being a beginner in differential geometry/topology, I'm not sure if the issue can be characterized in terms of that. 
