What is the rotation axis and rotation angle of the composition of two rotation matrix in $\mathbb{R}^{3}$ I was told in class that a rotation matrix is defined by a rotation
angle and rotation axis, if we call the rotation axis $v$ and take
a basis of $\mathbb{R}^{3}=\{v\}\bigoplus\{v\}^{\perp}$ then the
matrix is similar by an orthogonal matrix to a matrix of the form
$$\begin{pmatrix}\cos\theta & -\sin\theta\\
\sin\theta & \cos\theta\\
 &  & 1
\end{pmatrix}$$ 
I asked my self the following question: If I rotate in the $xy$ plain
(i.e. rotation axis is $z$) in angle $\theta$, and then rotate in
the $yz$ plain  (i.e. rotation axis is $x$) in angle $\varphi$
, what rotation matrix I get ? 
I tried multiplying the corresponding matrices but that did not produce
anything useful, I can't also thing of a vector $v\in\mathbb{R}^{3}$that
is invariant under the composition...
What is the rotation axis, and the rotation angle of these two compositions
? Help is appreciated!
 A: I assume you want the quick-and-ready answer Gibbs was teaching his maritime-inclined spherical-law-of-cosines Yale students in the 1880s.  With the advent of Pauli matrices for the fundamental rep of SU(2), this reduced to the standard group composition law of SU(2) ,  albeit the final answer is always a mess in terms of the original variables.
A rotation by angle θ around an axis $\hat{e}$ is represented by a Gibbs   vector $\vec{f}=\hat{e} \tan (\theta/2)$. So, for your first rotation, $ \vec{f}=\hat{z} \tan (\theta/2)$, and for your second one, $\vec{g}=\hat{x} \tan (\phi/2)$. 
The composition of the two rotations then amounts to 
$$
\frac{\vec{f}+\vec{g}-\vec{f}\times\vec{g}}{1-\vec{f}\cdot \vec{g}} ~.
$$
In your specific case, the dot product in the denominator vanishes, while the cross product in the numerator is in the pure y direction, which might illustrate why the eigenvector with eigenvalue 1 of your 3×3 matrix product appeared messy. 


*

*In any case, (Olinde Rodrigues, 1840), the axis boils down to the (un-normalized!) half-angle vector
$$
\hat{x} \sin\phi/2 \cos \theta/2+\hat{z} \cos\phi/2 \sin \theta/2 -\hat{y}\sin \phi/2 \sin\theta/2 . 
$$ 


Nevertheless, the combined rotation angle $\gamma$ is much simpler, as you see from the Pauli-matrix WP expression, namely a degenerate spherical law of cosines (the spherical analog of the Pythagorean theorem),
$$
\cos \gamma/2 = \cos\theta/2 \cos \phi/2 ,
$$
basically admiralty stuff. Cf. this answer, and this chapter.

To sum up, your new rotation axis is, indeed, left unchanged by the succesion of your two rotations,
$$\begin{pmatrix} 1 &  \\
&\cos\phi & -\sin\phi\\
 &\sin \phi  & \cos \phi
\end{pmatrix} \begin{pmatrix}\cos\theta & -\sin\theta\\
\sin\theta & \cos\theta\\
 &  & 1
\end{pmatrix} \begin{bmatrix}\cos\theta/2 \sin\phi/2\\
-\sin\theta/2   \sin\phi/2\\
 \sin\theta/2   \cos\phi/2
\end{bmatrix} =\begin{bmatrix}\cos\theta/2 \sin\phi/2\\
-\sin\theta/2   \sin\phi/2\\
 \sin\theta/2   \cos\phi/2
\end{bmatrix}  .$$ 


*

*Reality check: Take $\theta=\phi=\pi/2$ and watch $(1,-1,1)\mapsto (1,1,1)\mapsto(1,-1,1)$ in two steps.

A: When composing two rotations, it is useful to know that a rotation about $\alpha$ about an axis $\ell$ can be written as the composition of two reflections in planes containing $\ell$, the first being chosen arbitrarily and the second being at an (oriented) angle $\frac\alpha2$ with respect to the first. Now in the composition of $4$ reflections you get, you can make your choices so that the second and third planes of reflection (the second reflection for the first rotation and the first reflection for the second rotation) are both equal to the unique plane passing through the two axes. Then poof!, those second and third reflections annihilate each other, and you are left with the composition of the first and the fourth reflection, which is a rotation with axis the intersection of those planes, and angle twice the angle between those planes.
If you want to calculate the axis and angle in terms of the original angles, formulas get a bit complicated (even for very easy choices of initial axes as in the question), but such is life, the concrete answer isn't really very easy to write down or remember.
A: The axis of a rotation of a rotation matrix is its fixed point set.  so probably the easiest way to solve this problem is to multiply the two matrices and compute the fixed point set. this can be done by hand (a bit tedious) or with, say,mathematica in a couple of minutes
