Why does $A^TA=I, \det A=1$ mean $A$ is a rotation matrix?

I know if $A^TA=I$, $A$ is an orthogonal matrix. Orthogonal matrices also contain two different types: if $\det A=1$, $A$ is a rotation matrix; if $\det A=-1$, $A$ is a reflection matrix.

My question is: what is the relationship between the determinant of $A$ and rotation/reflection. Can you explain why $\det A=\pm 1$ means $A$ is a rotation/reflection from the geometric perspective?

EDIT: I think the following questions need be figured out first.

• Do rotation matrices only exist in 2D and 3D space? That is: for any dimensional matrix, as long as it is orthogonal and with determinant as 1, the matrix represents a rotation transformation, right? Note the orthogonality and determinant are applicable to arbitrary dimensional matrices.
• What is the most fundamental definition of a rotation transformation?
• Since an orthogonal matrix preserves length and angle, can we say an orthogonal matrix represents a "rigid body" transformation? "Rigid body" transformation contains two basic types: rotation and reflection?
• Do you mean in 2x2 matrices, or in general? Sep 28, 2011 at 3:44
• @Juan S, I know the rotation/reflection is valid in 2D and 3D. They should also exist in higher dimensional space, right? If yes, I mean nxn matrices. Sep 28, 2011 at 6:22
• If $A^TA=I$ and $A$ is a square matrix, i.e. has equally many rows and columns, then $A$ is an orthogonal matrix. But there are non-square matrices with orthonormal columns. If you take away some columns from an orthogonal matrix, the non-square matrix you're left with satisfies $A^TA=I$ but is not an orthogonal matrix. Sep 28, 2011 at 20:44
• @Michael: thanks, but non-square matrices don't have determinant:) I understand what you mean, of course, we consider square matrices herein. Sep 29, 2011 at 1:13

Others have raised some good points, and a definite answer really depends what kind of a linear transformation do we want call a rotation or a reflection.

For me a reflection (may be I should call it a simple reflection?) is a reflection with respect to a subspace of codimension 1. So in $\mathbf{R}^n$ you get these by fixing a subspace $H$ of dimension $n-1$. The reflection $s_H$ w.r.t. $H$ keeps the vectors of $H$ fixed (pointwise) and multiplies a vector perpendicular to $H$ by $-1$. If $\vec{n}\perp H$, $\vec{n}\neq0$, then $s_H$ is given by the formula

$$\vec{x}\mapsto\vec{x}-2\,\frac{\langle \vec{x},\vec{n}\rangle}{\|\vec{n}\|^2}\,\vec{n}.$$

The reflection $s_H$ has eigenvalue $1$ with multiplicity $n-1$ and eigenvalue $-1$ with multiplicity $1$ with respective eigenspaces $H$ and $\mathbf{R}\vec{n}$. Thus its determinant is $-1$. Therefore geometrically it reverses orientation (or handedness, if you prefer that term), and is not a rigid body motion in the sense that in order to apply that transformation to a rigid 3D body, you need to break it into atoms (caveat: I don't know if this is the standard definition of a rigid body motion?). It does preserve lengths and angles between vectors.

Rotations (by which I, too, mean simply an orthogonal transformations with $\det=1$) have more variation. If $A$ is a rotation matrix, then Adam's calculation proving that the lengths are preserved, tells us that the eigenvalues must have absolute value $=1$ (his calculation goes through for a complex vectors and the Hermitian inner product). Therefore the complex eigenvalues are on the unit circle and come in complex conjugate pairs. If $\lambda=e^{i\varphi}$ is a non-real eigenvalue, and $\vec{v}$ is a corresponding eigenvector (in $\mathbf{C}^n$), then the vector $\vec{v}^*$ gotten by componentwise complex conjugation is an eigenvector of $A$ belonging to eigenvalue $\lambda^*=e^{-i\varphi}$. Consider the set $V_1$ of vectors of the form $z\vec{v}+z^*\vec{v}^*$. By the eigenvalue property this set is stable under $A$: $$A(z\vec{v}+z^*\vec{v}^*)=(\lambda z)\vec{v}+(\lambda z)^*\vec{v}^*.$$ Its components are also stable under complex conjugation, so $V_1\subseteq\mathbf{R}^n$. It is obviously a 2-dimensional subspace, IOW a plane. It is easy to guess and not difficult to prove that the restriction of the transformation $A$ onto the subspace $V_1$ is a rotation by the angle $\varphi_1=\pm\varphi$. Note that we cannot determine the sign of the rotation (clockwise/ccw), because we don't have a preferred handedness on the subspace $V$.

The preservation of angles (see Adam's answer) shows that $A$ then maps the $n-2$ dimensional subspace $V^\perp$ also to itself. Furthermore, the determinant of $A$ restricted to $V_1$ is equal to one, so the same holds for $V_1^\perp$. Thus we can apply induction and keep on splitting off 2-dimensional summands $V_i,i=2,3\ldots,$ such that on each summand $A$ acts as a rotation by some angle $\varphi_i$ (usually distinct from the preceding ones). We can keep doing this until only real eigenvalues remain, and end with the situation: $$\mathbf{R}^n=V_1\oplus V_2\oplus\cdots V_m \oplus U,$$ where the 2D-subspaces $V_i$ are orthogonal to each other, $A$ rotates a vector in $V_i$ by the angle $\varphi_i$, and $A$ restricted to $U$ has only real eigenvalues.

Counting the determinant will then show that the multiplicity of $-1$ as an eigenvalue of $A$ restricted to $U$ will always be even. As a consequence of that we can also split that eigenspace into sums of 2-dimensional planes, where $A$ acts as rotation by 180 degrees (or multiplication by $-1$). After that there remains the eigenspace belonging to eigenvalue $+1$. The multiplicity of that eigenvalue is congruent to $n$ modulo $2$, so if $n$ is odd, then $\lambda=+1$ will necessarily be an eigenvalue. This is the ultimate reason, why a rotation in 3D-space must have an axis = eigenspace belonging to eigenvalue $+1$.

From this we see:

1. As Henning pointed out, we can continuously bring any rotation back to the identity mapping simply by continuously scaling all the rotation angles $\varphi_i,i=1,\ldots,m$ continuously to zero. The same can be done on those summands of $U$, where $A$ acts as rotation by 180 degrees.
2. If we want to define rotation in such a way that the set of rotations contains the elementary rotations described by Henning, and also insist that the set of rotations is closed under composition, then the set must consist of all orthogonal transformations with $\det=1$. As a corollary to this rotations preserve handedness. This point is moot, if we defined a rotation by simply requiring the matrix $A$ to be orthogonal and have $\det=1$, but it does show the equivalence of two alternative definitions.
3. If $A$ is an orthogonal matrix with $\det=-1$, then composing $A$ with a reflection w.r.t. to any subspace $H$ of codimension one gives a rotation in the sense of this (admittedly semi-private) definition of a rotation.

This is not a full answer in the sense that I can't give you an 'authoritative' definition of an $n$D-rotation. That is to some extent a matter of taste, and some might want to only include the simple rotations from Henning's answer that only "move" points of a 2D-subspace and keep its orthogonal complement pointwise fixed. Hopefully I managed to paint a coherent picture, though.

• I'm thinking any "rotation matrix" ought to be decomposable as a product of modified identity matrices with cosines and sines in all the proper places... and orthogonal matrices with negative determinant could be reflections or improper rotations... Sep 30, 2011 at 0:16
• @J.M. Thanks for the edits. My only excuse is the lateness of the hour. Some people (numerists and engineers) use Givens rotations as building blocks. I would like to include all transformations that rotate any 2D-subspace and keep its orthogonal complement pointwise fixed to the collection of elementary rotations. A Givens rotation would insist the 2D-subspace to be spanned by any two vectors of the standard basis. If we allow arbitrary compositions of such transformations the end result is the same. Sep 30, 2011 at 4:08
• @J.M.isapoormathematician: Group of Isometries is generated by reflections so you can definitely express them as combinations of reflections.
– MSIS
Jan 24, 2020 at 23:07

This depends on how we want to define "rotation" in the first place. Some people prefer a narrow definition where the only things that qualify as "rotations" are things that can be expressed as a $(2+n)\times(2+n)$ block matrix $$\pmatrix{\pmatrix{\cos\theta&\sin\theta\\-\sin\theta&\cos\theta}&0\\0&I}$$ with respect to some orthogonal basis. Under this definition there are $4\times 4$ matrices that are orthogonal and have determinant 1 but are not rotations - for example, $$\pmatrix{0.6&0.8&0&0\\-0.8&0.6&0&0\\0&0&0&1\\0&0&-1&0}$$

But one might also say that a "rotation" is any matrix $A$ such that there is a continuous family of matrices $(A_t)_{0\le t\le 1}$ such that $A_0=I$, $A_1=A$ and $A_t$ is always orthogonal. This captures the idea of a gradual isometric transformation away from a starting point. Such a definition immediately tells us that the determinant of a rotation must be 1 (because the determinant is continuous), but it is harder to see that we get all orthogonal matrices with determinant 1 this way.

I don't have a quick proof of the latter, but I imagine that it can be done fairly elementarily by induction on the dimension, first a series of rotations to make the first column fit, then sort out the remaining columns recursively working within the orthogonal complement of the first column.

• thanks. But I don't understand your 4 by 4 matrix example. The 'rotation' is not defined only in 2D and 3D space, right? Of course, the geometric meaning of a rotation in higher dimensional space is not intuitive. But any orthogonal matrix with determinant as 1 in arbitrary dimensional space can be called a rotation matrix, right? Sep 28, 2011 at 11:23
• One might also say that "rotations" are exponentials (in the Lie group sense) of members of the Lie algebra $so_n$ (which in this case can be identified as the skew-symmetric matrices). The member of the Lie algebra represents an "infinitesimal rotation" which generates a one-parameter group containing the given transformation. You can use the Spectral Theorem to show that every member of $SO_n$ can be obtained in this way. Sep 28, 2011 at 18:30
• @Robert Israel: Interesting! I'm not familiar with Lie algebra. But I do know that any rotation matrix can be expressed as an exponential of a skew-symmetric matrix. This actually is the axis-angle representation of a rotation. It may also known as "Rodrigues' rotation formula". If from this perspective, given a rotation matrix $R$, it can be written as $R=\mathrm{exp}(\Omega)$ with $\Omega$ as an skew-symmetric matrix. Then substituting $R=\mathrm{exp}(\Omega)$ into $\det R=\mathrm{exp}(\mathrm{tr}\ln R)$ gives $\det R=1$. Sep 29, 2011 at 1:23
• @Shiyu, Henning: I think it can be intuitive if one thinks of $\exp\Omega$ as $\lim\limits_{n\to\infty}(I+\frac1n\Omega)^n$. Basically, $I+\frac1n\Omega$ is a small transformation that moves each vector a little bit in a direction orthogonal to itself, so that (to first order) the vector's length is preserved. Then $\exp\Omega = \lim\limits_{n\to\infty}(I+\frac1n\Omega)^n$ means do a lot of these little length-preserving transformations one after another — so the net transformation is also length-preserving, and you end up with a rotation.
– user856
Sep 29, 2011 at 7:33
• @Henning: admittedly it wasn't quite as trivial as I remembered. Sep 30, 2011 at 2:54

The reason why it is called a rotation matrix is because lengths and angles are preserved.

Consider any $$v \in \mathbb{R}^n$$. Now consider the $$\ell_2$$ norm of $$Av$$: $$\|Av\|_2^2=v'A'Av = v'Iv=\|v\|_2^2.$$

Hence, the Euclidean length is invariant under an orthonormal linear transformation $$A$$. Furthermore, for any $$u, v \in \mathbb{R}^n,$$ we have the following equivalence in the inner products: $$\langle Au, Av \rangle_2 = u' A'A v = u'Iv = u'v = \langle u, v \rangle_2.$$

Note that since the range of cosine is $$[-1,1]$$, we may simply define the cosine of two vectors in a Euclidean space to be the normalized inner product as:

$$\cos \phi := \frac{\langle u, v \rangle_2}{\|u\|_2 \|v\|_2}$$

Given this definition and the preceding, it's easy to see that angles are preserved as well.

• That's not why it's a rotation: it's also true for reflections. Sep 28, 2011 at 5:53
• @Adam: thanks. However, if the length and angle are preserved, then the matrix is called a rotation matrix? Sep 28, 2011 at 11:12
• @Adam: can we interpret it in the following way? Because the length and angle are preserved, the orthogonal transformation is a rigid body transformation, which only contains rotation and reflection transformation? So an orthogonal matrix is either a rotation or reflection transformation? Sep 28, 2011 at 11:16
• @Robert: I didn't really pay attention to the "reflection" bit above because the only difference between the two is -1. Take any rotation matrix and multiply by -1: voila, you have a reflection matrix. They're essentially the same thing but for the latter the direction of the vectors is flipped. In particular, for any vector v parallel to the rotation axis (under rotation matrix A) we have Av = v, whereas for any vector v parallel to the reflection axis (under reflection matrix B = -A) we have Bv = -v (for either case notice that the angle between the vector and its transformation is 0).
Sep 29, 2011 at 6:52
• @Shiyu: Sure, you can view them as rigid body transformations, but just consider the directionality change I mentioned above for the reflection case. Note that both are isometries: distances between any two points are preserved (hence rotation/reflection -- nothing is being "stretched" or "shrunk").
Sep 29, 2011 at 7:04

I think John Stillwell's book Naive Lie Theory pretty much summarises the required points in this case.

He starts off by stating that it follows from the Cartan-Dieudonne theorem that a rotation about $O$ in $\mathbb{R^2}$ or $\mathbb{R^3}$ is a linear transformation that preserves length and orientation. He then goes on to justify that a transformation preserves length iff it preserves the inner product. By defining the rotation criterion in $\mathbb{R^n}$ as you have stated, it is shown that $AA^T=I\iff A$ preserves the inner product for a square matrix $A$ of order $n$. The two solutions $\det(A)=1$ and $\det(A)=-1$ occur accordingly as $A$ preserves orientation or not.

A perfect answer for your question can be found in the freely available paper: "A Disorienting Look at Euler’s Theorem on the Axis of a Rotation" By Bob Palais, Richard Palais, and Stephen Rodi.

In the paper, a linear algebra proof is discussed, that specifically deals with the relation between reflection, proper rotation, and the sign of the determinant. The proof is constructive in the sense that explicitly tackles the issue of invariant vectors compared to vectors that get reversed. As bonuses, Euler's E478 proof is recast into modern notation. A Topological proof is mentioned, as well as a Differential Geometry Lie Theory one.

Two reflections through a point $R_1R_2$ make a rotation $S$:

$$R_1R_2 = S$$

and $$det(AB) = det(A)*det(B) = ±1$$

You could just write out the components to confirm that this is so -- a much more interesting way to understand things, however, is to write down the condition as:

$$A^TIA=I$$

The idea is that the matrix $$A$$ preserves the identity quadratic form -- note that $$I$$ is a quadratic form here and not a linear transformation, as this is the transformation law for quadratic forms ($$A^TMA$$ instead of $$A^{-1}MA$$).

The hyperconic section corresponding to the identity quadratic form is the unit sphere -- thus the orthogonal transformations are all those that preserve the unit sphere. Another way of putting this is that $$(Ax)^TI(Ay)=x^TA^TIAy=x^TIy$$, i.e. the Euclidean dot product $$I$$ is preserved by $$A$$. This is equivalent to preserving the unit sphere, because the unit sphere is determined by the dot product on the given space.

What sort of transformations preserve the unit sphere?

The reason this is a good way of understanding things is that there are plenty of other "dot products" you can define. One elementary one from physics is the Minkowski dot product in special relativity, $$\mathrm{diag}(-1,1,1,1)$$ -- the corresponding quadric surface is a hyperboloid, and the transformations that preserve it, forming the Lorentz group, are boosts (skews between time and a spatial dimension), spatial rotations and reflections.

As for discriminating between rotations and reflections, suppose we define rotations in a completely geometric way -- for a matrix to be a rotation, all its eigenvalues are either 1 or in pairs of unit complex conjugates.

What do the eigenvalues of orthogonal matrices look like? For each eigenvalue, you need $$\overline{\lambda}\lambda=1$$, i.e. all the eigenvalues are unit complex numbers. If a complex eigenvalue isn't paired with a corresponding conjugate, you will not get a real-valued transformation on $$\mathbb{R}^n$$. Meanwhile if an eigenvalue of -1 isn't paired with another -1 -- i.e. if there are an odd number of reflections -- you get a reflection. The orthogonal (or rather unitary) transformations that do not behave this way are precisely the rotations.

The similarity between unpaired unit complex eigenvalues and unpaired -1's is interesting, by the way -- when thinking about reflections, you might have gotten the idea that reflections are $$\pi$$-angle rotations in a higher-dimensional space -- like the vector was rotated through a higher-dimensional space and then landed on its reflection -- like it was a discrete snapshot of a process as smooth as any rotation.

Well, now you know what this higher-dimensional space is -- precisely $$\mathbb{C}^n$$. And the determinant of a unitary matrix also takes a continuous spectrum -- the entire unit circle. In this sense (among other senses) complex linear algebra is more "complete" than real linear algebra.