# Orthonormal Matrices-Intuition

Why is it, geometrically, that the row space is ALSO orthonormal? What exactly does the transpose LOOK like? Normally the row and column space are two separate things, but in the case of an orthonormal matrix, you have that the row space is actually the inverse of the column space, so also, what does an inverse space look like? I feel there should be a very good geometric interpretation, but I don't see it.

-

I think your question is non-trivial. One way to see this is to consider the same problem in infinite dimensions. Let $H$ be an infinite dimensional Hilbert space with o.n. basis $e_1,e_2,\dots$. Let $T$ be the operator $$T e_i = e_{i+1} .$$ Then in some sense you can say that the columns of the matrix representing $T$ are orthonormal. In other words, $T$ is an isometry.

But now consider the adjoint operator $T^*$. This corresponds to the transpose of $T$. But $T^*$ has a one-dimensional kernel, and hence it is not an isometry, and hence its columns are not orthonormal.

This means that any answer to your question must take into account the fact that the matrix is finite dimensional.

-

I think this is actually a good question. We can talk about two things: the geometric intuition of an orthogonal matrix and the relationship between row and column space. The latter is slightly more general.

In the first place, let $O(n) = \{A \in M_n(\mathbb{R}) : A^T A = 1\}$, and let $SO(n)$ be the restriction of $O(n)$ to matrices with determinant 1. Note that this basically gets rid of the $A$ with negative determinant, since the condition $A^TA = I$ implies \begin{gather*} 1 = \det(I_n) = \det(A^T A) = \det(A^T)\det(A) = \det(A)^2, \end{gather*} since the transpose does not affect the determinant (prove it or trust me, it's probably not the key idea on a first reading, though it is a consequence of and implies non-trivial things). Therefore $\det(A) = \pm 1$.

Always give yourself simple examples (and counter-examples!). For instance, take $$A = \begin{pmatrix} 1 & 0 \\ 0 & -1\end{pmatrix}.$$ This matrix is not in $SO(n)$, since $\det(A) \neq 1$. Geometrically, this matrix is a reflection across the $x$-axis. To see this, see what happens to the basis vectors $(1,0)$ and $(0,1)$. The $x$-coordinate (1,0) stays fixed, while $y \rightarrow -y$. Note that this does not preserve orientation. That is,

Flips aren't all bad though. They still preserve volume. Matrices in $SO(n)$, however, preserve both orientation and volume and can be viewed as rotations. Consider for instance the matrix $$R = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}$$ This is a rotation matrix in $\mathbb{R}^2$. Its characteristic polynomial is $\lambda^2 -2\lambda \cos\theta + 1$, and the discriminant of this polynomial is $b^2 - 4ac = -4\sin^2(\theta).$ Therefore, whenever $\theta \neq 0 (mod 2\pi)$, the eigenvalues are imaginary. The eigenvalues of an orthogonal matrix always have norm $1$, so they are of the form $e^{i\theta_k}$ for various $k$.

If you want to talk about row and column space, you're going to have to think about the adjoint map a bit, i.e. $A \rightarrow A^*$, where $A^*$ is the matrix which satisfies $(Au,v) = (u,A^*v)$ for every $u, v$ in the vector space. Here $( \cdot,\cdot)$ denotes the inner product on the space (say the Euclidean inner product $x \cdot y$).

It turns out (one can prove by straight linear algebra by expanding sums and re-arranging terms) that the matrix which satisfies precisely that condition is always the transpose of the matrix. One can then ask what the relationship between these matrices is, or how one can think of them.

One interpretation is that if $A$ is a map from the vector space $V$ to v.s. $W$, then $A^* : W^* \rightarrow V^*$. Here, $V^*$ denotes the dual space of $V$ (ditto $W$). It can be seen as the map on row vectors ('dual vectors') induced by the map on column vectors.

Note: the reason I call row vectors 'dual vectors' is because one can associate to any row a linear functional on the vector space: write the row as $v^T$ and act on vectors $u$ by $u \rightarrow v^Tu \in \mathbb{R}$.

If you want I can get into this, depends on how much of this is below/above your level.

-

I'm unsure of exactly what you expect for "looks" like, but $O$ is orthonormal precisely when $O^T O = O O^T = I$. It works in both orders of the multiplication because that's how inverses always work: if they invert on one side, they invert on the other as well. $O^T O = I$ is precisely the statement that the columns are orthonormal, and $O O^T = I$ is precisely the statement that the rows are orthonormal.

-

Why is it, geometrically, that the row space is ALSO orthonormal? What exactly does the transpose LOOK like? [...] I feel there should be a very good geometric interpretation, but I don't see it.

Just synthesizing the existing answers and making some tacit points explicit.

For definiteness, let $A$ be an $n \times n$ real matrix whose $i$th column is $A_i$, viewed as an element of $\mathbf{R}^{n} = \mathbf{R}^{n \times 1}$, and let $A^t$ denote the transpose of $A$, i.e., the $n \times n$ real matrix whose $i$th row is $A_i^t$, viewed as an element of the dual space $(\mathbf{R}^{n})^* = \mathbf{R}^{1 \times n}$.

As zibadawa timmy notes, the columns of $A$ are an orthonormal set in $\mathbf{R}^n$ (i.e., the matrix $A$ is "orthonormal" in the OP's terminology) if and only if $A^t A = I$. Since $A$ is a (finite-size) square matrix of full rank, $AA^t = I$ as well, which implies the columns of $A^t$ are an orthonormal set in $\mathbf{R}^n$. (It's not clear the OP would view this argument as "geometric", but these observations directly answer the OP's question as I understand it.)

Finite-dimensionality is crucial, as Stephen Montgomery-Smith notes; the right shift operator in (e.g.) the space $\mathbf{R}^\infty = \bigcup_n \mathbf{R}^n$ of all finite (i.e., eventually zero) real sequences may be represented by a matrix $A$ with orthonormal columns, but having the zero vector as first row.

Finally, as snarski notes, the column space of $A$ is, properly speaking, a subspace of $\mathbf{R}^n$, while the row space of $A$ lies in $(\mathbf{R}^n)^*$. However, there's a tacit choice of inner product on $\mathbf{R}^n$ in the original question (the columns of $A$ are "orthonormal"), and this defines a "natural" isomorphism from $\mathbf{R}^n$ to $(\mathbf{R}^n)^*$, namely the transpose operator exchanging column and row matrices. Under this identification, the adjoint map $A^*:(\mathbf{R}^n)^* \to (\mathbf{R}^n)^*$, represented by the transpose $A^t$, is itself orthogonal (inner product preserving), so its matrix also has orthonormal columns. This is tantamount to saying $A$ has orthonormal rows.

-

A real orthogonal matrix is geometrically a rotation, a reflection or a permutation of the axes. For example, the matrix M = $\begin{pmatrix} cos(t) & -sin(t)\\ sin(t) & cos(t)\\ \end{pmatrix}$ rotates any vector through an angle of t. $M^{-1}$ undoes what M did, which is to say, it rotates every vector back through the angle (-t) and leaves it where it started. The matrix $\begin{pmatrix} 1 & 0\\ 0& -1\\ \end{pmatrix}$ reflects a vector across the x axis and its inverse reflects it back. This matrix can also be expressed in terms of sines and cosines if you wish.

The matrix $\begin{pmatrix} 0 & 1\\ 1& 0\\ \end{pmatrix}$ maps the x-axis into the y-axis and the y-axis into the x-axis. It inverse does the opposite. It also can be expressed in terms of sines and cosines.

To be sure this is right, just compute what each matrix does to a sample of vectors; figure out what its inverse is, and see that it undoes what M did to your sample.

-