orientation difference between two triangles in 3D space Lets there be 2 sets of 3 points in 3D space, representing 2 congruent isosceles triangles. The apexes of both triangles are located at the point (0,0,0). How do I calculate the difference between the orientation of the two triangles in terms of roll yaw and pitch? That is, by how much would I have to rotate one triangle in order for it to be in the exact  same location as the other one?
 A: Let the first triangle be $\triangle ABC$ and the second triangle be $\triangle A B' C' $ where $A = (0,0,0)$, then we want to relate the vertices through a rotation.  So, we have
$B' = R B $
$C' = R C $
we still need a third independent vector to specify the $3 \times 3$ matrix $R$.  That third vector can conveniently be choosen as the cross product between $B'$ and $C'$ and between $B$ and $C$, and we have from the properties of a rotation matrix that
$ B' \times C' = R B \times R C = R (B \times C) $
Putting all of the above in matrix format
$\begin{bmatrix} B' && C' && B' \times C' \end{bmatrix} = R \begin{bmatrix} B && C && B \times C \end{bmatrix} $
From which it follows that
$ R = \begin{bmatrix} B' && C' && B' \times C' \end{bmatrix} \begin{bmatrix} B && C && B \times C \end{bmatrix}^{-1} $
This gives the complete rotation $R$ from $\triangle ABC $ to $\triangle A B'C'$.
Now suppose the triangle being rotated is initially (at $t = 0$) at $\triangle A B C $ and finally (at $t = T$) at $\triangle A B' C' $, and you want to find the coordinates of the triangle vertices at an instant $t, 0 \le t \le T$, then you need to decompose the overall rotation matrix $R$ into the axis-angle format (which you can also call the Rodrigues Rotation Matrix format).
Any rotation matrix, can be put in the form
$ R = \mathbf{aa}^T + (I - \mathbf{aa}^T ) \cos \theta  + S_a \sin \theta $
where
$ S_a = \begin{bmatrix} 0 && - a_z && a_y \\ a_z && 0 && - a_x \\ -a_y && a_x && 0 \end{bmatrix} $
One can show based on this expression (which is the Rodrigues Rotation Matrix Formula) that the angle of rotation satisfies
$ \text{trace}(R) = R_{11} + R_{22} + R_{33} =  1 + 2 \cos \theta $
From which you can solve for $\theta$ (the overall rotation angle).  Now we need to find the axis of rotation $\mathbf{a}$.  Careful consideration of the off-diagonal entries of the above matrix $R$, leads to the following formulas
$ a_x = \dfrac{ R_{32} - R_{23} }{ 2 \sin \theta }$
$ a_y = \dfrac{ R_{13} - R_{31} }{ 2 \sin \theta }$
$ a_z = \dfrac{ R_{21} - R_{12} } {2 \sin \theta } $
Having found the axis of rotation $\mathbf{a}$ and the total angle of rotation $\theta$ (which corresponds to $t = T$ ), and assuming the triangle rotates at a constant rate, the angle of rotation at $ t $ where $0 \le t \le T$ is given by
$ \theta_1 = \bigg( \dfrac{t}{T} \bigg) \theta $
We can now compute the rotation matrix at $ t $, it is the same as before, but with the angle $\theta$ replaced with $\theta_1$
$ R(t) = \mathbf{aa}^T + (I - \mathbf{aa}^T ) \cos \theta_1  + S_a \sin \theta_1 $
Having found the rotation matrix $R(t)$, we can now compute the rotated triangle vertices at the instant $t$.
$ B'(t) = R(t) B $
$ C'(t) = R(t) C $
Of course, vertex $A$ is unaffected by the rotation and stays at $(0,0,0)$.
