What does it mean for two matrices to be orthogonal?

Firstly, please bear in mind that I do not have much knowledge about linear algebra. Secondly, I am not asking about some mathematical definitions but rather a more physical definition of this mathematical meaning.

So, the problem is that I can understand the meaning of orthogonality between two vectors, they are just "lines" perpendicular to each other, but I can not physically perceive what orthogonality means for matrices.(like a $$3\times 3$$ one). I mean if vectors are like lines in space, those being orthogonal is an easy concept to visualize. But how to visualize two orthogonal matrices?

I think that this is easier to explain to me if you first explain how does a matrix look like in space (visualization):

For example, for a $$2 \times 2$$, is it the area that its two contained vectors form in space? If it is that, then two orthogonal matrices are simply those two areas that are perpendicular to each other. If not, give me an analogous explanation.*

• Orthogonality of two vectors does not mean they are tangent to each other. Commented May 2, 2015 at 14:03
• Sorry,i meant perpendicular.I will edit Commented May 2, 2015 at 14:10
• @Mattos you understood the answers?, I have same doubt but not yet Commented Nov 25, 2018 at 7:27

There are two possibilities here:

1. There's the concept of an orthogonal matrix. Note that this is about a single matrix, not about two matrices. An orthogonal matrix is a real matrix that describes a transformation that leaves scalar products of vectors unchanged. The term "orthogonal matrix" probably comes from the fact that such a transformation preserves orthogonality of vectors (but note that this property does not completely define the orthogonal transformations; you additionally need that the length is not changed either; that is, an orthonormal basis is mapped to another orthonormal basis). Another reason for the name might be that the columns of an orthogonal matrix form an orthonormal basis of the vector space, and so do the rows; this fact is actually encoded in the defining relation $A^TA = AA^T = I$ where $A^T$ is the transpose of the matrix (exchange of rows and columns) and $I$ is the identity matrix.

Usually if one speaks about orthogonal matrices, this is what is meant.

2. One can indeed consider matrices as vectors; an $n\times n$ matrix is then just a vector in an $n^2$-dimensional vector space. In such a vector space, one can then define a scalar product just as in any other vector space. It turns out that for real matrices, the standard scalar product can be expressed in the simple form $$\langle A,B\rangle = \operatorname{tr}(AB^T)$$ and thus you can also define two matrices as orthogonal to each other when $\langle A,B\rangle = 0$, just as with any other vector space.

To imagine this, you simply forget that the matrices are matrices, and just consider all matrix entries as components of a vector. The two vectors then are orthogonal in the usual sense.

It is not common to say that two matrices are orthogonal to each other, but rather one speaks of a matrix being an orthogonal matrix.

Formally, a matrix $A$ is called orthogonal if $A^TA = AA^T = I$. In other words, the columns of the matrix form a collection of orthogonal (and normed vectors); if you take two distinct columns they are orthogonal as vectors. (You could also consider rows.)

"Physically" an orthogonal matrix corresponds to a distance preserving linear transformation (such as a rotation) of the space.

• Just to clarify something,if we have say a 2x2 matrix,this means that like a vector is a "line" in space,then the matrix is an area in space that is formed by the vectors that are inside the matrix? Commented May 2, 2015 at 14:28
• @LandosAdam not quite. The two vectors in the matrix can be thought of as the axis of a (new) coordinate system. The transformation I mention take the "standard" coordinate vectors to these new coordinate vectors.
– quid
Commented May 2, 2015 at 14:32
• So,what exactly is a matrix if we are to visualize it in the same way as we visualize a vector in space? Commented May 2, 2015 at 14:33
• An ordered list of vectors.
– quid
Commented May 2, 2015 at 14:36
• Yes,but this does not give me a visualization.I meant,like a vector is a line in space,what can we say about a matrix?What does it represent geometrically?Is it just there to transform or there is a graphical representation for it? Commented May 2, 2015 at 14:39

$\newcommand{\Reals}{\mathbf{R}}$Let $m$ and $n$ be positive integers. The set $\Reals^{m \times n}$ of all real $m \times n$ matrices is "essentially" the space $\Reals^{mn}$ of all real vectors with $mn$ components: Just put the entries of a matrix into a single column in some specified order. The ordinary dot product in $\Reals^{mn}$ allows us to speak of Euclidean geometry in the space of $m \times n$ matrices. In particular, we may speak of two matrices being "orthogonal" if their dot product is zero.

It turns out that the dot product of two $m \times n$ real matrices $A$ and $B$ has a simple formula, $\operatorname{tr}(A^{T}B)$. (In words, multiply the transpose of $A$ by $B$, then add up the diagonal entries of the resulting square matrix.)

However, as quid says, there's a completely different definition of "orthogonal" as a property of a single $n \times n$ matrix, which is "If $u$ and $v$ are vectors in $\Reals^{n}$, then $u \cdot v = 0$ if and only if $Au \cdot Av = 0$." This turns out to be equivalent to "Multiplication by $A$ preserves all concepts of Euclidean geometry", or "The columns of $A$ form an orthonormal basis of $\Reals^{n}$", among others.

If someone comes to you and says, "Let $A$ and $B$ be two orthogonal matrices, ...", quid's interpretation is probably what they mean. But if lives depend on it, clarify with the asker. :)

• I understand all this.But this is a mathematical explanation which i already know.Don't get me wrong,it is a good answer but not the answer i am seeking.I just want to visualize what a matrix is like i visualize what a vector is,and then visualize what orthogonal matrices are. Commented May 2, 2015 at 14:46
• One also should never forget to ask with respect to which inner product. ;-) Nice answer.
– quid
Commented May 2, 2015 at 14:54
• @LandosAdam: No offense taken if this answer isn't helpful to you. :) But it sounds as if the picture you're seeking boils down to, e.g., "A (non-zero) $2 \times 2$ real matrix is an arrow in four-dimensional space." That said, it may help your intuition to consider matrix groups, e.g., $SL(2, \mathbf{R})$ (a hyperboloid-like hypersurface in $\mathbf{R}^{4}$) or $SO(3)$ (a "twisted product" of an ordinary sphere and a circle in $\mathbf{R}^{9}$ that casts an obvious "shadow" in $\mathbf{R}^{6}$ if you look only at the first two columns). Commented May 2, 2015 at 15:07

I think this is the crux of your question:

first explain how does a matrix look like in space(visualization)

Matrices are harder to visualize than vectors.

Here's one way to attack the problem. In the context you care about, a matrix is really a way to move vectors around in the plane - it's a way to represent a linear transformation. (I won't say more now since you may know that. If you don't, I can elaborate.) The unit square with corners $(0,0), (1,0), (0,1), (1,1)$ always moves to a parallelogram formed by two vectors with their tails at the origin. That parallelogram is one way to visualize the matrix. Then the matrix is orthogonal when that parallelogram is a unit square - the two edge vectors from the origin are orthogonal (and also of length 1).

I will focus on $$n\times n$$ matrices since the $$n\times m$$ generalization is fairly straightforward. Define a matrix $$V=\begin{bmatrix} v_{11} & v_{12} & &... & & v_{1n} \\ v_{21} & v_{22} & & & & . \\ .& & . & & & .\\ . & & &. & &.\\ v_{n1} & . & . & .& . & v_{nn} \end{bmatrix}$$

For notational purposes, denote $$v_{ab}=(\vec{v}_b)_a$$, i.e. the $$a$$-th component of the vector $$\vec{v}_b$$. This way we can write $$V$$ as a matrix whose columns are vectors $$\vec{v}_1,\vec{v}_2,..,\vec{v}_n$$. Note that this changes nothing in the forthcoming conclusion. The matrix can then be written as:$$V=\begin{bmatrix} (\vec{v}_1)_1 & (\vec{v}_2)_1 & &... & & (\vec{v}_n)_1 \\ (\vec{v}_1)_2 & (\vec{v}_2)_2 & & & & . \\ .& & . & & & .\\ (\vec{v}_1)_n & . & . & .& . & (\vec{v}_n)_n \end{bmatrix}=\begin{bmatrix} . & . & .& .& .& .\\ . & . & . & . & . & . \\ \vec{v}_1& \vec{v}_2 & . & . & . & \vec{v}_n\\ . & . & . &. & . &.\\ .& . & . & . &. & . \end{bmatrix}$$ This way, computing $$V^T V$$ gives: $$V^T V=\begin{bmatrix} . & . & \vec{v}_1^T& .& .& .\\ . & . & \vec{v}_2^T & . & . & . \\ .& . & . & . & . & .\\ . & . & . &. & . &.\\ . & . & \vec{v}_n^T & .& . &. \end{bmatrix} \begin{bmatrix} . & . & .& .& .& .\\ . & . & . & . & . & . \\ \vec{v}_1& \vec{v}_2 & . & . & . & \vec{v}_n\\ . & . & . &. & . &.\\ . & . & . & .& . &. \end{bmatrix}=\begin{bmatrix} |\vec{v}_1|^2 & \vec{v}_1\cdot\vec{v}_2 & .& . & \vec{v}_1\cdot\vec{v}_n\\ \vec{v}_1\cdot\vec{v}_2 & |\vec{v}_2|^2 & . & . & . \\ .& . & . & . & .\\ . & . & . & . &.\\ \vec{v}_1\cdot\vec{v}_n & . & .& . &|\vec{v}_n|^2 \end{bmatrix}$$ where we have defined $$\vec{v}_i\cdot \vec{v}_j=\sum_a (\vec{v}_i)_a(\vec{v}_j)_a$$ and used $$\vec{v}_j\cdot\vec{v}_j=\vec{v}_j\cdot\vec{v}_i$$.

Now, if $$V$$ is an orthogonal matrix, meaning $$V^T V=\mathbb{1}_{n\times n}$$, it means that $$\vec{v}_i\cdot \vec{v}_j=\delta_{ij}$$ where I have defined $$\delta_{ij}=1$$ when $$i=j$$ and $$\delta_{ij}=0$$ when $$i\neq j$$.

Conclusion: An $$n\times n$$ orthogonal matrix V is a matrix that consists of columns (or rows) which are orthonormal vectors, i.e. they are unit vectors that are normal to each other. These vectors can form a basis for an n-dimensional vector space since they are linearly independent*. $$\boxed{V^TV=\mathbb{1}_{n\times n}\ \longleftrightarrow\ \vec{v}_i\cdot \vec{v}_j=\delta_{ij}}$$

Extra: the determinant of $$V$$ gives the oriented volume (in n-dimensions) of a hypercube whose sides are formed by the vectors which are either the rows or the columns (since $$\det{V}=\det{V^T}$$) of $$V$$.

*Linear independence can be seen from the fact that taking the det of both sides of $$V^TV=\mathbb{1}_{n\times n}$$ gives $$(\det{V})^2=1$$.