What does it mean for two matrices to be orthogonal?

Question

Firstly, please bear in mind that I do not have much knowledge about linear algebra. Secondly, I am not asking about some mathematical definitions but rather a more physical definition of this mathematical meaning.

So, the problem is that I can understand the meaning of orthogonality between two vectors, they are just "lines" perpendicular to each other, but I can not physically perceive what orthogonality means for matrices.(like a $3\times 3$ one). I mean if vectors are like lines in space, those being orthogonal is an easy concept to visualize. But how to visualize two orthogonal matrices?

I think that this is easier to explain to me if you first explain how does a matrix look like in space (visualization):

For example, for a $2 \times 2$, is it the area that its two contained vectors form in space? If it is that, then two orthogonal matrices are simply those two areas that are perpendicular to each other. If not, give me an analogous explanation.*

Answer 1

There are two possibilities here:

1. There's the concept of an orthogonal matrix. Note that this is about a single matrix, not about two matrices. An orthogonal matrix is a real matrix that describes a transformation that leaves scalar products of vectors unchanged. The term "orthogonal matrix" probably comes from the fact that such a transformation preserves orthogonality of vectors (but note that this property does not completely define the orthogonal transformations; you additionally need that the length is not changed either; that is, an orthonormal basis is mapped to another orthonormal basis). Another reason for the name might be that the columns of an orthogonal matrix form an orthonormal basis of the vector space, and so do the rows; this fact is actually encoded in the defining relation $A^TA = AA^T = I$ where $A^T$ is the transpose of the matrix (exchange of rows and columns) and $I$ is the identity matrix.

   Usually if one speaks about orthogonal matrices, this is what is meant.

2. One can indeed consider matrices as vectors; an $n\times n$ matrix is then just a vector in an $n^2$-dimensional vector space. In such a vector space, one can then define a scalar product just as in any other vector space. It turns out that for real matrices, the standard scalar product can be expressed in the simple form $$\langle A,B\rangle = \operatorname{tr}(AB^T)$$ and thus you can also define two matrices as orthogonal to each other when $\langle A,B\rangle = 0$, just as with any other vector space.

   To imagine this, you simply forget that the matrices are matrices, and just consider all matrix entries as components of a vector. The two vectors then are orthogonal in the usual sense.

Answer 2

It is not common to say that two matrices are orthogonal to each other, but rather one speaks of a matrix being an orthogonal matrix.

Formally, a matrix $A$ is called orthogonal if $A^TA = AA^T = I$. In other words, the columns of the matrix form a collection of orthogonal (and normed vectors); if you take two distinct columns they are orthogonal as vectors. (You could also consider rows.)

"Physically" an orthogonal matrix corresponds to a distance preserving linear transformation (such as a rotation) of the space.

Answer 3

$\newcommand{\Reals}{\mathbf{R}}$Let $m$ and $n$ be positive integers. The set $\Reals^{m \times n}$ of all real $m \times n$ matrices is "essentially" the space $\Reals^{mn}$ of all real vectors with $mn$ components: Just put the entries of a matrix into a single column in some specified order. The ordinary dot product in $\Reals^{mn}$ allows us to speak of Euclidean geometry in the space of $m \times n$ matrices. In particular, we may speak of two matrices being "orthogonal" if their dot product is zero.

It turns out that the dot product of two $m \times n$ real matrices $A$ and $B$ has a simple formula, $\operatorname{tr}(A^{T}B)$. (In words, multiply the transpose of $A$ by $B$, then add up the diagonal entries of the resulting square matrix.)

However, as quid says, there's a completely different definition of "orthogonal" as a property of a single $n \times n$ matrix, which is "If $u$ and $v$ are vectors in $\Reals^{n}$, then $u \cdot v = 0$ if and only if $Au \cdot Av = 0$." This turns out to be equivalent to "Multiplication by $A$ preserves all concepts of Euclidean geometry", or "The columns of $A$ form an orthonormal basis of $\Reals^{n}$", among others.

If someone comes to you and says, "Let $A$ and $B$ be two orthogonal matrices, ...", quid's interpretation is probably what they mean. But if lives depend on it, clarify with the asker. :)

Answer 4

I think this is the crux of your question:

first explain how does a matrix look like in space(visualization)

Matrices are harder to visualize than vectors.

Here's one way to attack the problem. In the context you care about, a matrix is really a way to move vectors around in the plane - it's a way to represent a linear transformation. (I won't say more now since you may know that. If you don't, I can elaborate.) The unit square with corners $(0,0), (1,0), (0,1), (1,1)$ always moves to a parallelogram formed by two vectors with their tails at the origin. That parallelogram is one way to visualize the matrix. Then the matrix is orthogonal when that parallelogram is a unit square - the two edge vectors from the origin are orthogonal (and also of length 1).

Answer 5

I will focus on $n\times n$ matrices since the $n\times m$ generalization is fairly straightforward. Define a matrix $$V=\begin{bmatrix} v_{11} & v_{12} & &... & & v_{1n} \\ v_{21} & v_{22} & & & & . \\ .& & . & & & .\\ . & & &. & &.\\ v_{n1} & . & . & .& . & v_{nn} \end{bmatrix}$$

For notational purposes, denote $v_{ab}=(\vec{v}_b)_a$, i.e. the $a$-th component of the vector $\vec{v}_b$. This way we can write $V$ as a matrix whose columns are vectors $\vec{v}_1,\vec{v}_2,..,\vec{v}_n$. Note that this changes nothing in the forthcoming conclusion. The matrix can then be written as:$$V=\begin{bmatrix} (\vec{v}_1)_1 & (\vec{v}_2)_1 & &... & & (\vec{v}_n)_1 \\ (\vec{v}_1)_2 & (\vec{v}_2)_2 & & & & . \\ .& & . & & & .\\ (\vec{v}_1)_n & . & . & .& . & (\vec{v}_n)_n \end{bmatrix}=\begin{bmatrix} . & . & .& .& .& .\\ . & . & . & . & . & . \\ \vec{v}_1& \vec{v}_2 & . & . & . & \vec{v}_n\\ . & . & . &. & . &.\\ .& . & . & . &. & . \end{bmatrix}$$ This way, computing $V^T V$ gives: $$V^T V=\begin{bmatrix} . & . & \vec{v}_1^T& .& .& .\\ . & . & \vec{v}_2^T & . & . & . \\ .& . & . & . & . & .\\ . & . & . &. & . &.\\ . & . & \vec{v}_n^T & .& . &. \end{bmatrix} \begin{bmatrix} . & . & .& .& .& .\\ . & . & . & . & . & . \\ \vec{v}_1& \vec{v}_2 & . & . & . & \vec{v}_n\\ . & . & . &. & . &.\\ . & . & . & .& . &. \end{bmatrix}=\begin{bmatrix} |\vec{v}_1|^2 & \vec{v}_1\cdot\vec{v}_2 & .& . & \vec{v}_1\cdot\vec{v}_n\\ \vec{v}_1\cdot\vec{v}_2 & |\vec{v}_2|^2 & . & . & . \\ .& . & . & . & .\\ . & . & . & . &.\\ \vec{v}_1\cdot\vec{v}_n & . & .& . &|\vec{v}_n|^2 \end{bmatrix}$$ where we have defined $\vec{v}_i\cdot \vec{v}_j=\sum_a (\vec{v}_i)_a(\vec{v}_j)_a$ and used $\vec{v}_j\cdot\vec{v}_j=\vec{v}_j\cdot\vec{v}_i$.

Now, if $V$ is an orthogonal matrix, meaning $V^T V=\mathbb{1}_{n\times n}$, it means that $$\vec{v}_i\cdot \vec{v}_j=\delta_{ij}$$ where I have defined $\delta_{ij}=1$ when $i=j$ and $\delta_{ij}=0$ when $i\neq j$.

Conclusion: An $n\times n$ orthogonal matrix V is a matrix that consists of columns (or rows) which are orthonormal vectors, i.e. they are unit vectors that are normal to each other. These vectors can form a basis for an n-dimensional vector space since they are linearly independent*. $$\boxed{V^TV=\mathbb{1}_{n\times n}\ \longleftrightarrow\ \vec{v}_i\cdot \vec{v}_j=\delta_{ij}}$$

Extra: the determinant of $V$ gives the oriented volume (in n-dimensions) of a hypercube whose sides are formed by the vectors which are either the rows or the columns (since $\det{V}=\det{V^T}$) of $V$.

*Linear independence can be seen from the fact that taking the det of both sides of $V^TV=\mathbb{1}_{n\times n}$ gives $(\det{V})^2=1$.