What are the requirements for a rotation matrix?

Generally speaking, what are the necessary and sufficient properties of a matrix to make it a rotation matrix?

Is det(A) = 1 enough?

-

No, not by far. The matrix needs to be orthogonal, which means that $A^tA=I$ where $A^t$ is the transposed matrix -- and then it also has to have determinant 1.

(You can think of orthogonality by considering how the matrix acts on the standard basis vectors -- since they were orthogonal to each other and had length 1 before the rotation, this must also be true after the rotation. But the vectors after-the-rotation are just the columns of $A$, and $A^tA=I$ is then a compact way to take the dot product of each pair of new vectors and see if they come out right, all in one operation).

-
That $A^tA=I$ doesn't guarantee it's an orthogonal matrix unless it's also a square matrix, i.e. the number of rows is the same as the number of columns. – Michael Hardy Sep 28 '11 at 3:12
It is easy to understand why orthogonal matrices preserve the vector length. Can you explain why $\det A=1$ means rotation and $\det A=-1$ means reflection? – Shiyu Sep 28 '11 at 3:22
@Shiyu, not right off the cuff. Intuitively it's because $SO(n)$ is connected, but I don't have a quick demonstration of that ready. Also, I don't even think I'd be comfortable calling something like $\pmatrix{0.6&0.8&0\\-0.8&0.6&0\\0&0&-1}$ a "reflection". – Henning Makholm Sep 28 '11 at 3:44
@Shiyu: Since there is such an operation as an improper rotation/rotoreflection, which you'll get if you multiply a rotation and reflection matrix, I'm not comfortable calling orthogonal matrices with a determinant of $-1$ as "reflections" either. – J. M. Sep 28 '11 at 9:36

Not it is not. A rotation matrix is a square matrix with orthonormal columns and determinant 1. The set of all $n\times$n such matrices is commonly referred to as the special orthogonal group, and is denoted $SO(n)$.

-

In $\mathbb{R}^3$, one can prove that the set of all rotation matrices is precisely $SO_3(\mathbb{R})$.

Assuming this fact, a rotation matrix $A$ with respect to the standard basis would then have the properties

1. $AA^{T} = I$
2. $\langle Ax, Ay \rangle = \langle x,y \rangle$, where $\langle,\rangle$ is just the standard inner product in $\mathbb{R}^3$ and $x,y$ any vectors in $\mathbb{R}^3$.
3. $||Ax|| = ||x||$ for any vector $||x||$.

In fact the properties above hold not just for the rotation matrices but for matrices in $O_3$ (and in general $O_n$ ) as well.

It remains now to prove why the rotation matrices are exactly the matrices of $SO_3$. To prove it you need at least two facts:

1. Every linear operator on a real vector space $V$ of odd dimension has a (real) eigenvalue.

2. If $W$ is a subspace of $V$ stable the restriction of a linear operator $T$ to $W$, then $W^{\perp}$ is stable too.

-