How can I break down a rotation of known amount around a known axis into two rotations of unknown amounts around known axes? I have a vector that's been rotated a known amount about a known axis. I would like to break this rotation down into two separate rotations around known, linearly independent axes where the amounts I need to rotate about each of these axes is unknown and need to be calculated.
I'm using matrices for my rotations at the moment but I'm happy to use quaternions if that's an easier way to calculate this.
I would like a closed form solution but I think there's a good chance that one does not exist.
All this takes place in 3-space.
Thanks,
 A: Let ${\mathbf u}=(x,y,z)$ ($\|\mathbf u\|=1$) be the axis of rotation and $\theta$ be the angle of rotation. For the two component rotations, denote the axes and angle of rotations by ${\mathbf u}_i=(x_i,y_i,z_i)$ ($\|{\mathbf u}_i\|=1$) and $\theta_i$ for $i=1,2$. Essentially, you are going to solve
$$
c + sx{\mathbf i} + sy{\mathbf j} + sz{\mathbf k} =
(c_1 + s_1x_1{\mathbf i} + s_1y_1{\mathbf j} + s_1z_1{\mathbf k})
(c_2 + s_2x_2{\mathbf i} + s_2y_2{\mathbf j} + s_2z_2{\mathbf k}),
$$
where $c=\cos\frac\theta2,\,s=\sin\frac\theta2,\,c_1=\cos\frac{\theta_1}2,\,s_1=\sin\frac{\theta_1}2,\,c_2=\cos\frac{\theta_2}2,\,s_2=\sin\frac{\theta_2}2$. (Note: rotation 2 is applied first, and then rotation 1. If you want the other way round, interchange the indices 1 and 2.) Multiply out the RHS and collect terms, we get
\begin{align}
c &= c_1c_2 - s_1s_2(x_1x_2+y_1y_2+z_1z_2),\tag{1}\\
sx &= c_1s_2 x_2 + c_2s_1 x_1 + s_1s_2 (y_1z_2 - z_1y_2),\tag{2}\\
sy &= c_1s_2 y_2 + c_2s_1 y_1 + s_1s_2 (z_1x_2 - x_1z_2),\tag{3}\\
sz &= c_1s_2 z_2 + c_2s_1 z_1 + s_1s_2 (x_1y_2 - y_1x_2).\tag{4}
\end{align}
Equations (2)-(4) give
$$
\begin{pmatrix}
x_2&x_1&y_1z_2 - z_1y_2\\
y_2&y_1&z_1x_2 - x_1z_2\\
z_2&z_1&x_1y_2 - y_1x_2
\end{pmatrix}
\begin{pmatrix}
c_1s_2\\c_2s_1\\s_1s_2
\end{pmatrix}
=
\begin{pmatrix}
sx\\sy\\sz
\end{pmatrix}.
$$
Solving it, we get
$$
\begin{pmatrix}
c_1s_2\\c_2s_1\\s_1s_2
\end{pmatrix}
=
\frac{1}{1 - ({\mathbf u}_1\cdot{\mathbf u}_2)^2}
\begin{pmatrix}
1-x_1({\mathbf u}_1\cdot{\mathbf u}_2)&1-y_1({\mathbf u}_1\cdot{\mathbf u}_2)&1-z_1({\mathbf u}_1\cdot{\mathbf u}_2)\\
1-x_2({\mathbf u}_1\cdot{\mathbf u}_2)&1-y_2({\mathbf u}_1\cdot{\mathbf u}_2)&1-z_2({\mathbf u}_1\cdot{\mathbf u}_2)\\
y_1z_2-z_1y_2&z_1x_2-x_1z_2&x_1y_2-y_1x_2
\end{pmatrix}
\begin{pmatrix}
sx\\sy\\sz
\end{pmatrix}.
$$
Substitute this result into (1), we obtain also $c_1c_2$. Hence
\begin{align*}
\theta_1 = \begin{cases}
2\operatorname{atan2}(c_1c_2, s_1c_2)&\ \text{ if not } c_1c_2=s_1c_2=0,\\
2\operatorname{atan2}(c_1s_2, s_1s_2)&\ \text{ otherwise},
\end{cases}\\
\theta_2 = \begin{cases}
2\operatorname{atan2}(c_1c_2, c_1s_2)&\ \text{ if not } c_1c_2=c_1s_2=0,\\
2\operatorname{atan2}(s_1c_2, s_1s_2)&\ \text{ otherwise},
\end{cases}
\end{align*}
where $\mathtt{atan2}$ is the quadrant-aware arctangent function.
A: If I interpret your question correctly, given
$\alpha$, $\hat{u}$, $\hat{v}$, $\hat{w}$
you are looking for $\beta$, $\gamma$ such that
rotating an arbitrary vector, $\vec a$, by $\gamma$ about $\hat w$,
then by $\beta$ about $\hat v$ 
is the same as rotating $\vec a$ by $\alpha$ about $\hat u$.
This cannot be done in general.
Consider the situation where $\alpha=90^\circ$, 
$\hat u=\hat z$, $\hat v=\hat y$ and $\hat w=\hat x$ 
and consider the action on the vector $\vec a=\hat x$.
Rotating $\vec a=\hat x$ about $\hat u=\hat z$ by $\alpha=90^\circ$ 
will carry it into $\hat y$.
However, rotating $\vec a=\hat x$ about $\hat w=\hat x$ by any amount 
leaves it invariantly equal to $\hat x$.  Rotation then about
$\hat v=\hat y$ by any amount will leave it within the $xz$-plane and cannot carry it into $\hat y$.  
Furthermore, unless some special a-priori relationship 
exists between the given $\alpha$, $\hat{u}$, $\hat{v}$, and $\hat{w}$, 
the choice of $\beta$ and $\gamma$ has insufficient degrees of freedom
to be able to solve the desired conditions.
