Do matrices really rotate and stretch vectors or is that definition incorrect?

I come from the applied math and statistics world, but I was talking to my friend who comes from the pure math and number theory world—in particular Galois representation theory.

I mentioned something about the confusing definition of "matrices" in textbooks. Many textbooks talk about a matrix as the solution to a linear system of equations, or other abstract descriptions. The definition that I have always found useful, was the sense that matrices rotate and scale vectors through linear transformations.

Now, coming from the pure math side, my friend said that this definition was not accurate. I am trying to paraphrase some of his comments, but he said that in higher dimensions, matrices can stretch and rotate vectors only locally. He also said that it depends on what vectors the matrix acts upon.

His response threw me for a loop and I was trying to understand how to resolve his statements. First, is my understanding of matrices incorrect?

It is fine if this idea of rotating and stretching a vector is incorrect, but I am not sure what the alternative definition of a matrix would be. Like when my friend says that a matrix may only stretch/rotate a vector locally, I am trying to think of a practical example of this kind of phenomena.

I will follow up with my friend on what he means, but I was hoping someone could set me straight on how I should think of a matrix, or the definition of a matrix.

• The singular value decomposition of a real $m \times n$ matrix $A$ is worth reading about. Feb 12 at 10:03
• @littleO I am familiar with the SVD. Can you explain your comment a bit more. Feb 12 at 16:37
• FWIW your definition of matrices, involving stretching and rotating vectors, sounds like the kind of thing a physicist would say. It's known that physicists and mathematicians have some differing ideas about vectors and matrices (I asked about this once and I think I just recently saw a similar question about the definition of tensors) so it may help to get clear about whether each of you is talking about the physical or mathematical idea of a vector (or about something entirely different). Feb 12 at 18:54
• @krishnab The SVD expresses a real $m \times n$ matrix $A$ as $A = U \Sigma V^T$ where $U$ and $V$ are orthogonal matrices and $\Sigma$ is diagonal. In my mind, this is the closest we can come to describing a generic matrix as stretching and rotating input vectors. Feb 12 at 20:41
• It's hard to say what your friend was getting at from this. It sounds like he was being a bit provocative! Perhaps his point was that matrices do not always have a sensible associated linear map, or that not all linear maps are so geometrical (eg high dimensions/weird base fields). Or perhaps he was objecting to you calling "matrices rotate and scale vectors" a definition - I would personally say that's a description/intuition, not a definition. Your understanding is most likely fine for most applied uses of matrices, but there are indeed other useful perspectives for pure maths! Feb 12 at 22:52

I don't know what your friend was getting at, but a (square) matrix—meaning an $$n\times n$$ array of elements in some field $$k$$—can always be interpreted as representing a linear transformation on $$k^n$$, which in turn can be interpreted as rotating and stretching vectors. That's definitely a true statement.

In general linear maps $$k^n\to k^m$$ of course correspond to $$m\times n$$ matrices, which are generally nonsquare. In that case it makes a bit less sense to compare vectors between the two spaces. However, I'd argue the intuition is still correct: if $$m, for example, you can consider $$k^m$$ as the first $$m$$ dimensions of $$k^n$$ and pad your matrix with zeros to be $$n\times n$$.

Your friend was either mistaken or talking about some other, looser concept of matrices. But I would solidly say that "matrices" are inherently tied to linear maps on finite dimensional vector spaces, at least in >99% of modern usage.

• Non-square matrices are pretty common, and I don't think it really makes sense to call them rotations/stretching. It's a little unclear if you're claiming they fall in the 1% or 99% Feb 12 at 19:28
• @cobbal I should have said $m\times n$. It makes less sense to "stretching and rotating" when the domain and codomain are different, but I'd argue the idea is morally the same. I'll add to my answer. Feb 12 at 20:56
• Eigenvectors are not rotated, only stretched (which is not in contrast with your statement, as you say "which in turn can be interpreted as rotating and stretching vectors." (where I suppose the word "and" means one, or the other, or both :) ): but maybe that's one of the aspects where the "pure math side" gentleman was in disagreement with the statement that the "definition was not accurate". Feb 13 at 16:02
• @JanStuller maybe, but if I were being as pedantic as possible I would also call rotation by $0$ degrees a rotation. All hell breaks loose when a reflection appears though (joking of course) Feb 14 at 6:09
• @JanStuller But can be "stretched" by $e^{i\theta}$, and that's a rotation if you are flexible with how you interpret the basis. :-) Feb 14 at 13:01

When you ask what a matrix "really is", you are asking the wrong question. A matrix is, at its heart, just a big block of numbers arranged in a rectangular grid. That's it.

The important question is: what can you do with a matrix? That's a much more interesting story! In fact matrices can be used in many different ways:

• They can store data. In many practical applications, this is their primary use. The "tax table" that says how much American taxpayers owe the Internal Revenue Service every year is just a matrix. So are population and census tables. A list of prices of a set of $$n$$ items at a grocery store over a period of $$m$$ weeks can be arranged as an $$n \times m$$ matrix. If you have such a matrix, you can look up information in it.
• A matrix can store the coefficients of a system of linear equations in $$n$$ variables. By performing row operations on such a matrix, we can solve the system of equations.
• An $$n \times m$$ matrix $$A$$ can be used to define a linear transformation $$T: \mathbb R^m \to \mathbb R^n$$ via the definition $$T(\vec v) = A\vec v$$. Here, the right-hand side of the equation is just matrix-vector multiplication.

But this is just scratching the surface! Here are some additional uses of matrices:

• An $$n \times m$$ matrix $$A$$ can also be used to define a linear transformation $$S: \mathbb R^n \to \mathbb R^m$$ (notice the difference between this and the item above) via the definition $$S(\vec w) = \left((\vec w)^\top A \right)^\top$$. This is a different transformation than the standard transformation $$T$$ defined above, although they are closely related.
• An $$n \times m$$ matrix $$A$$ can also be used to define a linear transformation $$L: \mathbb R^{m \times d} \to \mathbb R^{n \times d}$$ via the definition $$L(B) = AB$$ (left-multiplication by $$A$$). Again, this is a different transformation than the previous examples: whereas $$T$$ and $$S$$ take column vectors as inputs and produce column vectors as outputs, $$L$$ takes a matrix as input and produces another matrix as output. So matrices can be the inputs and outputs of a linear transformation, as well as the "code" that determines a linear transformation itself!
• An $$n \times m$$ matrix $$A$$ can also be used to define a linear transformation $$R: \mathbb R^{d \times n} \to \mathbb R^{d \times m}$$ via the definition $$R(B) = BA$$ (right multiplication by $$A$$). Again, this is a different transformation than the previous examples: $$R$$ (like $$L$$) acts on a space of matrices, but has a different kernel, image, etc.
• If an $$n \times n$$ matrix $$A$$ happens to be invertible, then we can use it to define a linear transformation $$C: \mathbb R^{n \times n} \to \mathbb R^{n \times n}$$ via $$C(B) = A^{-1} B A$$ ("conjugation by $$A$$"). The theory of diagonalization is basically the question: given $$B$$, when does there exist a matrix $$A$$ such that $$C(B)$$ is diagonal?
• If $$f:\mathbb R^n \to \mathbb R^m$$ is a differentiable function, we can form the matrix of partial derivatives $$A_{ij} = \partial y_i / \partial x_j$$, which provides a local linear description of $$f$$, in the same way that the standard derivative of single-variable calculus gives a local linear description of a function $$f:\mathbb R \to \mathbb R$$.

And on, and on, and on. The moment of inertia tensor in Physics is a matrix that describes the distribution of mass throughout a 3-dimensional body; the Hessian matrix of second partial derivatives is a matrix that describes (sort of) the concavity of a function of many variables in different directions; etc., etc.

Some of the specific applications above can be thought of as special cases of other, more general applications also in the same list, and I suppose you could try to arrange the list of applications in some kind of hierarchical system and try to identify one "primary" meaning that contains all of the other ones. But to do so you would probably have to generalize things so much that you would end up with nothing more specific than "a matrix is a big block of numbers arranged in a rectangular grid".

• Nice answer! I appreciate that your list can't keep growing forever but an application that sticks out to me is "there is a correspondence between $m \times n$ matrices and bilinear forms on $\Bbb R^m \times \Bbb R^n$". Also in the first linear maps example, the fact that it goes the other way too (each linear map comes from a matrix). These two respective examples are particularly meaningful when talking about Jacobian/Hessian matrices! Feb 12 at 22:45
• @mweiss thanks for this answer. Your point is valid, that there is a difference between what a matrix is--a big block of numbers--versus what a matrix can do. For example when I work in statistics, I use matrices to project a vectors onto a line that minimizes the total error. When I solve PDEs numerically, I usually solve large sparse linear systems over and over again. But I wonder how would you explain what a matrix is to a high schooler who has never seen a matrix before--or performed the operations you describe. How can you explain the idea yet preserve the accuracy of the definition? Feb 13 at 21:10
• @krishnab. I would honestly just say "A matrix is a big block of numbers, and there are lots of things you can do with them." For a newcomer I would stick with using a matrix to store & lookup data, and maybe talk about applications where adding matrices or multiplying by a scalar has a meaningful interpretation. Row-reduction and matrix-vector multiplication would come later. Feb 13 at 21:44

I'm not sure what distinction the word locally is trying to convey. Perhaps your friend was talking about matrix fields, ie continuously varying matrices? Though that doesn't seem particularly relevant to number theory.

The answer depends on how general stretching is. Specifically, you need to include shearing, something like $$(x, y) \mapsto (x, x + y)$$. A picture from wikipedia (not of that exact map):

It is a general fact that (the action of) any matrix in any dimension is a composition of:

• shears (elementary matrices)
• stretches (diagonal matrices, maybe included in above depending on definitions)
• inclusions (in coordinates, adding some $$0$$s; eg $$x \mapsto (x, 0)$$ including the $$x$$-axis in the plane)
• projections (maps forgetting some of the coordinates, eg $$(x, y) \mapsto x$$ projection the plane onto the $$x$$-axis)

The usual abstract form of this statement is:

Elementary matrices and diagonal matrices generate the general linear group (invertible matrices).

• Don't you also need rotations? Feb 12 at 8:22
• @Gerry a rotation is a composition of shears, eg $(x, y) \mapsto (x + y, y) \mapsto (x + y, -x) \mapsto (y, -x)$ is rotation by $90^\circ$ clockwise. Feb 12 at 8:31

In the “linear transformation” point of view, matrices represent linear transformations, by denoting where the unit vector in the $$x$$ direction goes in the first column and where the unit vector in the $$y$$ direction goes in the second. Of course for $$3$$ dimensions there are more basis vectors but the same applies.

Then, we could form the matrix like this:

$$\begin{bmatrix} a_{x_1} & a_{y_1} \\ a_{x_2} & a_{y_2} \\ \end{bmatrix}$$

Then, we could apply it to a vector using matrix-vector multiplication:

$$\begin{bmatrix} a_{x_1} & a_{y_1} \\ a_{x_2} & a_{y_2} \\ \end{bmatrix} \begin{bmatrix} x \\ y \\ \end{bmatrix}$$

Which would result in

$$\begin{bmatrix} xa_{x_1} + ya_{y_1} \\ xa_{x_2} + ya_{y_2} \\ \end{bmatrix}$$

This is the vector when you stretch, rotate, and scale it by the matrix.

It would look something like this (for an arbitrary transformation):

Grey: gridlines of the original plane

Blue: gridlines of the transformed plane

Here, we could form a matrix. Red: unit vector in the $$x$$ direction, and green: unit vector in the $$y$$ direction.

Then the matrix would look like this:

$$\begin{bmatrix} 3 & 1 \\ 0 & -2 \\ \end{bmatrix}$$

Matrices have many definitions; from coefficients of linear systems to arrays of numbers to linear transformations, and none of them are inherently “wrong” or “right”.

The linear transformations view is indeed a valid and IMO intuitive view of matrices. There is no “problem” in viewing them as such.

However, we have to carefully differentiate between definition and use. We define a matrix as a rectangular array of numbers. What we can do with it is much more fruitful - it can model differential equations, describe linear transformations, describe linear systems, and many more.

Summary: Matrices stretching and rotating vectors is not a fundamental definition. It is one of the many fruitful uses of rectangular arrays of numbers.

3Blue1Brown illustrates this in “Linear transformations and matrices”.

First of all: "stretching and rotating" is not a definition, is intuition. The intuition on stretching and rotation you have is ok, although it depends a little on what you mean. All functions rotate and stretch vectors, that is not really the interesting thing; the point and the special feature of linear transformations is that they rotate and stretch vectors in a sort of "uniform" and "rigid" way.

Like all intuition, one needs to question it and take it with a grain of salt. Let's consider the intuition "matrices stretch and rotate vectors in a rigid/uniform way". For example you might ask: does it mean that you can always write a matrix as a composition (i.e., product) of a pure rotation(/reflection) matrix and a pure stretching matrix? The answer (for square matrices) is indeed yes, and this is precisely the Polar decomposition of matrices. For rectangular matrices there are similar decompositions as well, like the singular value decomposition.

The fact that these decompositions exist is not trivial though, and it does not simply follow from the definition of matrix. In principle, different vectors could be "rotated" and "stretched" in different ways, with no correlation whatsoever.

Just another remark. A "pure stretching" matrix does not necessarily stretch vectors uniformly in all directions. For example, I would consider a diagonal matrix to be of "pure stretching" type. The matrix $$A=\left[\begin{array} 21&0\\0&2\end{array}\right]$$ transforms the vector $$e_1=(1,0)^T$$ to itself, it maps $$e_2=(0,1)^T$$ to $$2e_2$$, and sends the vector $$v=e_1+e_2$$ to $$w=e_1+2e_2$$. So, the vectors $$e_1$$ and $$e_2$$ are stretched by different amounts, and the vector $$v$$ is mapped to a vector which points in a different direction. In general, if the entries of the diagonal are distinct, the vectors parallel to the axis are stretched by different amounts depending on the corresponding element in the diagonal. And vectors that are not parallel to the axis, in principle are also a bit rotated.

To be a bit more precise on the vague terms I used above. A "pure stretching" matrix is the loose term I use for symmetric matrices in the case of real coefficients (resp. Hermitian matrices in the complex case), and it is essentially just a diagonal matrix up to an orthonormal change of basis. And "pure rotation" matrices are called orthogonal in the real case (resp. unitary in the complex case).

Final note. Not all maps matching your description are linear. Affine transformations also rotate and stretch vectors (in a uniform/rigid way). That is why yours can't be a definition.

• thank you for your insights here. Indeed, I think the word "intuition" is the correct one. My intent in the conversation was to give an example of some simple intuitive explanation for the matrix as an object--as if I was explaining a matrix to a non-mathematician. But my question seems to have stirred a lot of interest :). Feb 13 at 18:47
• I see :) This kind of "soft" questions on intuition behind some mathematical objects usually attract many answers. Also, sorry that I was pedantic at the beginning, I just needed to make the point that "rotating and stretching" can mean very different things, and it's worth trying to question this kind of heuristics and compare them with known theorems. Feb 13 at 20:57
• No problems. I was certainly being rather "loose" in my description, so thanks for setting me straight. Basically the true definition is much broader that my shorthand. Yeah, I suppose I used this loose description of a matrix as a shorthand to manage the complexity of so many abstract thoughts. Feb 13 at 21:27

You are quite close. One can always decompose a matrix using the singular value decomposition (SVD). For some matrix $$M$$ this is written as

$$M=U\Sigma V^*,$$ where $$V^*$$ means the conjugate transpose of $$V$$, $$U$$ and $$V$$ are unitary matrices and $$\Sigma$$ is a diagonal matrix. Unitary means $$VV^*=1\!\!1$$ and it is the complex equivalent of a rotation matrix.

In the case that $$M$$ is real-valued and a square $$(n\times n)$$ matrix, $$U$$ and $$V$$ can be chosen to be real, which means unitarity becomes orthogonality, i.e. $$VV^T=1\!\!1$$. So in that case $$U$$ and $$V$$ are rotation matrices (or reflections).

So this means that any real, square matrix can be seen as a rotation, a scaling and some other rotation. And maybe a reflection as well.

Consider the matrix $$A=\begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix}$$. In the Euclidean plane, this matrix is the matrix in the canonical basis of rotation with $$+\frac{pi}2$$. For example, $$D=(5,2)\mapsto D'=(-2,5)$$. But let $$u=(5,0)$$ and $$v=(2,3)$$ In the base $$\{u,v\}$$, the linear map $$f$$ such that $$f(u)=v$$ and $$f(v)=-u$$ also has the matrix $$A$$. But then we won't talk about rotation. Maybe that's why "he also said that it depends on what vectors the matrix acts upon."

You have to be precise, even during the superbowl;)