What are the requirements for a rotation matrix? Generally speaking, what are the necessary and sufficient properties of a matrix to make it a rotation matrix?
Is det(A) = 1 enough?
 A: Not it is not.  A rotation matrix is a square matrix with orthonormal columns and determinant 1.  The set of all $n\times $n such matrices is commonly referred to as the special orthogonal group, and is denoted $SO(n)$.
A: No, not by far. The matrix needs to be orthogonal, which means that $A^tA=I$ where $A^t$ is the transposed matrix -- and then it also has to have determinant 1.
(You can think of orthogonality by considering how the matrix acts on the standard basis vectors -- since they were orthogonal to each other and had length 1 before the rotation, this must also be true after the rotation. But the vectors after-the-rotation are just the columns of $A$, and $A^tA=I$ is then a compact way to take the dot product of each pair of new vectors and see if they come out right, all in one operation).
A: In $\mathbb{R}^3$, one can prove that the set of all rotation matrices is precisely $SO_3(\mathbb{R})$.
Assuming this fact, a rotation matrix $A$ with respect to the standard basis would then have the properties


*

*$AA^{T} = I$

*$\langle Ax, Ay \rangle = \langle x,y \rangle $, where $\langle,\rangle$ is just the standard inner product in $\mathbb{R}^3$ and $x,y$ any vectors in $\mathbb{R}^3$.

*$||Ax|| = ||x||$ for any vector $||x||$.


In fact the properties above hold not just for the rotation matrices but for matrices in $O_3$ (and in general $O_n$ ) as well.
It remains now to prove why the rotation matrices are exactly the matrices of $SO_3$. To prove it you need at least two facts:


*

*Every linear operator on a real vector space $V$ of odd dimension has a (real) eigenvalue.

*If $W$ is a subspace of $V$ stable the restriction of a linear operator $T$ to $W$, then $W^{\perp}$ is stable too.
