What are the basic types of orthogonal transformations of $\mathbb{R}^3$ and is there a calculus? It is well known that there are only two types of  orthogonal transformations of $\mathbb{R}^2$, namely rotations and reflections. Moreover, they have a nice calculus as outlined in https://en.wikipedia.org/wiki/Rotations_and_reflections_in_two_dimensions
By this I mean: composition of two reflections is a rotation. Rotation before/after reflection is reflection. Reflection after reflection is rotation. All transformations are assumed to fix the origin.  
My question is: What is the counterpart of this for $\mathbb{R}^3$? There are reflections and rotations. What is their "calculus"? Do we need other types? Any reference or link would be greatly appreciated!
 A: This is typically defined as the orthogonal group which is denoted $O(n)$ and can be defined as 
$$
O(n) \;\; =\;\; \{A \in M_n(\mathbb{R}) \; | \; AA^T = A^TA = I\}.
$$
These matrices have orthonormal columns and rows and preserve the Euclidean dot product in that if $Q \in O(n)$ then
$$
\langle Qx, Qy\rangle \;\; =\;\; x^TQ^TQy \;\; =\;\; x^Ty \;\; =\;\; \langle x,y\rangle.
$$
Similarly $O(n)$ also breaks up into two connected pieces, one containing the rotations and the other containing reflections (the group of rotations typically denoted as $SO(n)$ and called the special orthogonal group).  One can easily tell whether an orthogonal matrix is a reflection or a rotation by computing its determinant.  You can easily show that $\det Q = \pm 1$ for all $Q \in O(n)$.  Rotations are those matrices with $\det Q = 1$ and reflections are those with $\det Q = -1$.
The calculus part comes into play because both $O(n)$ and $SO(n)$ are smooth manifolds, and better yet are smooth manifolds with a compatible group structure making them both into Lie groups.  Therefore it makes sense to talk about smoothly transitioning from one rotation to another (although one cannot smoothly transition between a rotation to a reflection).
In the special case of $n=3$, there are some nice properties of orthogonal matrices.  One that's particularly i nice is that any orthogonal matrix can be represented by a product of fundamental rotations about each axis.  In particular we can define the matrices
\begin{eqnarray*}
R_z(\theta) & = & \left [ \begin{array}{ccc}
\cos \theta & - \sin \theta & 0 \\
\sin \theta & \cos \theta & 0 \\
0 & 0& 1 \\
\end{array} \right ] \\
R_y(\phi) & = & \left [ \begin{array}{ccc}
\cos \phi & 0 &  - \sin \phi \\
0 & 1& 0 \\
\sin \phi & 0 & \cos \phi  \\
\end{array} \right ] \\
R_x(\gamma) & = & \left [ \begin{array}{ccc}
1 & 0 & 0 \\
0 & \cos \gamma & -\sin \gamma \\
0 & \sin \gamma & \cos \gamma \\
\end{array} \right ] \\
\end{eqnarray*}
Then any orthogonal matrix can be written as a product of the above matrix for different values of $\theta, \phi, \gamma$ which are the roll, pitch, and yaw angles.
