Angle of rotation based on direction cosines I have a question which is bothering me for days! Suppose that we have a fixed frame $XYZ$ and a moving frame $xyz$ in 3D. The moving frame is orthonormal and is defined based on the fixed one using 9 direction cosines. For instance, the unit vector $x$ is $(l_1,m_1,n_1)$ where $l_1$, $m_1$ and $n_1$ are the cosines of the angles between $x$ and $X$, $Y$ and $Z$ respectively. Similarly, we have $y=(l_2,m_2,n_2)$ and $z=(l_3,m_3,n_3)$ which are also unit vectors.
My question is: At first the moving frame $xyz$ coincides $XYZ$. Then it rotates arbitrary to form a frame with known direction cosines. How can I calculate the angle of rotation of the moving frame around its $z$ axis based on the 9 direction cosines. In other words, how much the $x$-axis rotates around the $z$-axis?
Thanks a lot for saving me! 
 A: I didn't read your question closely enough. Below is the answer to how to find which axis the rotation is about and the angle of rotation about that axis. I'm leaving it, though, in case it is useful.

Set this form of the rotation matrix equal to your version. Solve for the axis vector $(u_x, u_y, u_z)$ and the angle of rotation $\theta$
A: The short answer is that there is not a single way to undo the rotation.
The basic concept is that mapping the attitude of one system (XYZ) to another, can be done in infinite number of ways using the rotation operators; such as angles-axis which is a single operation, or Euler angles which is what you seem to be asking for.
Euler angles are comprised of three different rotations; (there are six different sequences for Euler angles). The point is: if you change the order/sequence of these three rotations, but with same value for angles, you would end up with a different outcome.
So when you're asking how much x rotates around z, you should decide on how you want to continue the rest of rotation; is it going to rotation around the new x and then the new z? or is it going to rotate first around y and then z.
But if you decide on the whole process, let's say ZXY-Euler, and if you are looking for how to convert your rotation to a ZXY-Euler then you have to solve an equation system with 9 equations and 3 variables (good news), and there are singularities at certain angles (bad news!). Take a look at rotation matrices based on Euler angles to solve for Euler-anlges when you have your rotation matrix (3x3).
A: As already mentioned, a rotation in 3D Euclidean space can be defined in many ways,
which essentially boils down to consider the product of  three elemental rotations, either
relevant to the fixed system (extrinsic), or applied to the moving system (intrinsic).
Refer to this Wikipedia article.
In any case, once you have your transformation
$$
\left\{ \matrix{
  {\bf x} = l_{\,x} \,{\bf i} + m_{\,x} \,{\bf j} + n_{\,x} \,{\bf k} \hfill \cr 
  {\bf y} = l_{\,y} \,{\bf i} + m_{\,y} \,{\bf j} + n_{\,y} \,{\bf k} \hfill \cr 
  {\bf z} = l_{\,z} \,{\bf i} + m_{\,z} \,{\bf j} + n_{\,z} \,{\bf k} \hfill \cr}  \right.
$$
you know that the inverse is given by the transpose matrix, and it is clear
that the axis $X$, whose unit vector is $\bf i$, has the components $(l_x,l_y)$
in the $x,y$ plane, corresponding to an angle of 
$$
\alpha \;:\quad \cos \alpha  = {{l_{\,x} } \over {\sqrt {l_{\,x} ^{\,2}  + l_{\,y} ^{\,2} } }},\quad \sin \alpha  = {{l_{\,y} } \over {\sqrt {l_{\,x} ^{\,2}  + l_{\,y} ^{\,2} } }}
$$
which upon rotation (around $z$) alignes with $\bf x$, i.e.has an angle $\alpha=0$.
Thus the rotation you are looking for is $- \alpha$.
A: I'm going to answer the question you put in a comment, since it might be the problem you are really trying to solve (as opposed to the question you literally asked).
Assuming the frame use orthonormal coordinate systems, one approach is to use rotation matrices. 
This is especially simple when you have direction cosines of the rotated frame.
In your notation, the rotation matrix is
$$ \begin{pmatrix} l_1 & l_2 & l_3 \\
m_1 & m_2 & m_3 \\
n_1 & n_2 & n_3
\end{pmatrix}.$$
Take the $xyz$ coordinates (in the rotated frame) of any point,
put them in a column vector of numbers, and multiply the rotation matrix with the column vector. The result is the $XYZ$ coordinates of the point.
The "undo" rotation is given by the matrix
$$ \begin{pmatrix} l_1 & m_1 & n_1 \\
l_2 & m_2 & n_2 \\
l_3 & m_3 & n_3 
\end{pmatrix}.$$
Now, this approach does not tell you any angle or angles of rotation around any axis or sequence of axes. But it does allow you to rotate coordinates to and from the rotated frame as much as you like.
