What is an intuitive way to understand the dot product in the context of matrix multiplication?

I was trying to understand where it came from that each row in a matrix multiplication is a dot product, as in:

$$Ax = \left( \begin{array}{ccc} a_{1}^T \\ \vdots \\ a_m^T \end{array} \right)x = \left( \begin{array}{ccc} a_{1}^Tx \\ \vdots \\ a_m^T x \end{array} \right)$$

what is an intuitive explanation or interpretation that each row is a dot product of the vector x?

What I do understand is that $Ax$ encodes a linear transformation $T$. Consider a super simple example in 2 dimensions to explain what I do understand. I understand that $Ax = A [x_1 x_2] = T(v) = T(x_1 \hat i + x_2 \hat j) = x_1 T(\hat i) + x_2 T( \hat j)$. This makes me interpret intuitively that a multiplication by a matrix gives me a new vector that is composed of the same linear combination of the transformed basis vectors (or whatever vectors v is composed of)[source]. Furthermore one can easily see from this view where the multiplication of a matrix comes from:

$$Ax = \left[ \begin{array}{ccc} a_{11} & a_{12} \\ a_{21} & a_{22} \\ \end{array} \right] x = \left[ \begin{array}{ccc} T(\hat i)_1 & T(\hat j)_1\\ T(\hat i)_2 & T(\hat j)_2\\ \end{array} \right] \left[ \begin{array}{ccc} x_{1} \\ x_{2} \\ \end{array} \right] = x_1\left[ \begin{array}{ccc} T(\hat i)_1 \\ T(\hat i)_2 \\ \end{array} \right] + x_2 \left[ \begin{array}{ccc} T(\hat j)_1\\ T(\hat j)_2\\ \end{array} \right] = \left[ \begin{array}{ccc} T(\hat i)_1 x_1 + T(\hat j)_1 x_2\\ T(\hat i)_2x_2 + T(\hat j)_2 x_2\\ \end{array} \right]$$

where now its obvious why matrix multiplication is defined the way it is (because of linear transformations). Notice that the nice thing about this view is that one can interpret that each column of the matrix tells us how each basis vector changes. i.e. each column specifies how $\hat i$, $\hat j$ are transformed. Furthermore, the amount it used to be in the old vector is retained but now its in the new direction $T(\hat i)$ for the first coordinate. This for me is really intuitive and explains a lot of where matrix multiplication comes from.

However, if you notice this view reveals that each row $(Ax)_i = a_1^T x$ is a dot product of the initial array representation of the vector. This seems to me to not be a coincidence and that something deeper has to be going on. Usually dot products are related with projections so I was trying to understand if each coordinate of $(Ax)_i$ might actually be encoding how much the original $x$ is being projected into each row vector of $A$ (or possible something to do with the row space of $A$ i.e. $C(A^T)$ ). In an attempt to understand this I considered what each row means:

$$\left[ a_{i,1} \dots a_{i,m} \right] \left[ \begin{array}{ccc} x_1\\ \vdots\\ x_n \\ \end{array} \right] = \sum^n_{j=1} a_{ij} x_j$$

in the old interpretation I had of what a column of a matrix is (this time the matrix is 1 by m), it seems that the columns $a_{i,j}$ specifies how much some basis vector $e_i$ is transformed. However, I've had difficulties understanding beyond that what the significance of the dot product of $x$ with the rows of $A$ means. Does someone know how to interpret this or how to understand it at a conceptual level, similar to the way the interpretation I gave of what the columns of a matrix mean? Are we doing some transformation to the row space of $A$ or something like that?

• Did you ever end up fully understanding an intuitive reason for the dot-product interpretation? – 1110101001 Nov 14 '17 at 7:44
• @1110101001 I think I just gave up. However, I think there are some insights here: youtube.com/… however I don't think I fully digested it in a way that 100% made sense to me. If you find a way to explain it better please feel free to share it . – Pinocchio Nov 14 '17 at 16:51
• I read your question more carefully, deleted my old post, and threw up a new one more in line with what your asking. What it tells you is the row space of a (real) matrix is the orthogonal complement of its nul space.This actually turns out to be an important result when considered in the context of your numerical linear algebra question. – David Reed Nov 20 '17 at 22:56

We have $x = x_i e_i = x'_i e'_i$ where $e_i$ and $e'_i$ are bases related by nonsingular linear transformation. Note that ${e'}_i^T e'_j = g_{ij}$, where $g$ is invertible. Thus, ${e'}_i^T e_j x_j = {e'}_i^T e'_j x'_j = g_{ij}x'_j$ or $$x'_i = (g^{-1})_{ij}{e'}_j^T e_k x_k = (g^{-1})_{ij}{e'}_j^T x.$$ This gives us two good pieces of intuition. First, for a nonsingular linear transformation $A$ we can think of the elements of $A$ as being given by $$a_{ij} = (g^{-1})_{ik}{e'}_k^T e_j,$$ that is, by the dot product of a certain linear combination of the transformed basis vectors with the untransformed basis vectors. Second, to find the result of applying $A$ to $x$ we simply dot the same linear combination of the transformed basis vectors with the vector $x$.

For orthogonal transformations we find $g_{ij} = \delta_{ij}$ and so $$x'_i = {e'}_i^T e_j x_j = {e'}_i^T x \hspace{5ex}\textrm{and}\hspace{5ex} a_{ij} = {e'}_i^T e_j.$$

Note: We use Einstein's summation convention, $x = x^i e_i \equiv \sum_i x^i e_i$. For this problem the dual basis is $e^i = e_i^T$. The dual of $x$ is $x^T$, so $x_i e^i = x^i e_i^T$. We need not distinguish between $x_i$ and $x^i$ and so we write $x = x_i e_i$.

Example

Let $$A = \left(\begin{array}{cc}\cos\theta & \sin\theta \\ -\sin\theta & \cos\theta \end{array}\right).$$ Then $$\left(\begin{array}{cc}\cos\theta & \sin\theta \\ -\sin\theta & \cos\theta \end{array}\right) \left(\begin{array}{c}x \\ y\end{array}\right)$$ will give the components of $x$ in the new basis $e'_i$, where $[e_i]_j = \delta_{ij}$ is the standard basis. (This is a passive, rather than active, transformation.) It is straightforward to show that $e'_i = A^{-1}e_i = A^T e_i,$ so $$e'_1 = \left(\begin{array}{c}\cos\theta \\ \sin\theta\end{array}\right) \hspace{5ex}\textrm{and}\hspace{5ex} e'_2 = \left(\begin{array}{c}-\sin\theta \\ \cos\theta\end{array}\right).$$ One can then easily check that the elements of $A$ are given by $a_{ij} = {e'}_i^T e_j$. Note that, $$x'_1 = {e'}_1^T e_j x_j = \left(\begin{array}{cc}\cos\theta & \sin\theta\end{array}\right) \left(\begin{array}{c}x \\ y\end{array}\right)$$ and $$x'_2 = {e'}_2^T e_j x_j = \left(\begin{array}{cc}-\sin\theta & \cos\theta\end{array}\right) \left(\begin{array}{c}x \\ y\end{array}\right),$$ as expected.

It is easiest to see this in one dimension first.

Our goal is to show that any linear transformation $T: \mathbb{R}^n \rightarrow \mathbb{R}$ can be represented in the form $Tu = \beta^Tu$ for some $n$-dimensional vector $\beta$. Say that $u \in \mathbb{R}^n$; let $e_1, \ldots, e_n$ be the standard basis vectors for $\mathbb{R}^n$ (where $e_i$ has a $1$ in the $i$th position, and $0$'s elsewhere.) Then we can write $u = \sum_i u_i e_i$ where $u_i$ is the $i$-th coordinate of $u$. Since $T$ is linear, we have $$Tu =T \sum_i u_i e_i = \sum_i u_i Te_i = \sum_i u_i \beta_i,$$ where $\beta_i = Tb_i$. This means that with respect to the standard basis, $Tu = \beta \cdot u$ where $\beta$ is the vector $(\beta_1, \ldots, \beta_n)$. Thus every linear map from $\mathbb{R}^n$ to $\mathbb{R}$ can be represented by taking the dot product with a fixed vector.

Now for the multi-dimensional case: If $T: \mathbb{R}^n \rightarrow \mathbb{R}^m$ then $T$ is equivalent to the $m$-tuple of functions $(T_1, T_2, \ldots, T_m)$ where $T_jx$ is the $j$-th coordinate of $Tx$. Then for each $j$ there is a vector $\beta^j$ with $T_jx = \beta^j \cdot x$ and the result follows. (note that the $j$ in $\beta^j$ is just a superscript here meaning the $j$-th one, and doesn't have anything to with the $j$-th power.)

I think an intuitive way to think about matrix multiplication is to regard it as a combination of coordinate extraction and scalar multiplication. First, given any basis for a finite vector space, such as $\mathbb{R}^n$, we usually express any vector $\mathbf{v}=\sum_i c_i \mathbf{e}_i=[c_1,c_2,\dots,c_n]$ as an $n$-tuple of scalars. Given any vector $\mathbf{v}$, the functions that extracts the $i$-th coordinate $c_i,$ are linear functionals, and form a basis for the dual space. They can be thought of similar to projection maps.

Second, given any scalar $c$, the operation of multipliying a vector $\mathbf{v}$ by $c$ to produce $c\mathbf{v}$ is a linear transformation to $\mathbb{R}^n$. The composition of extracting the $i$-th coordinate of a vector $\mathbf{v}$ and then multiplying the scalar by another vector $\mathbf{e}_j$ as a linear transformation from $\mathbb{R}^n$ to $\mathbb{R}^m$ we denote by $T_{ij}.$ Now a matrix $\mathbf{A}$ with entries $a_{ij}$ is associated with the finite sum $\sum_{ij} a_{ij}T_{ij}$ as a linear transformation. Note that the identity map $I_n=\sum_i T_{ii}$ is the sum of projection maps $T_{ii}.$

We can think of this in two ways. First, the $i$-th row of the matrix $\mathbf{A}$ is associated with the composite map $\mathbf{v} \mapsto (\sum_j a_{ij} c_j)\mathbf{e}_i$ which is a dot product scalar multiplied by a basis vector. Second, the $j$-th column of the matrix $\mathbf{A}$ is associated with the composite map $\mathbf{v} \mapsto c_j(\sum_i a_{ij}\mathbf{e}_i)$ which is the scalar $c_j$, the $j$-th coordinate of $\mathbf{v}$, times the $j$-th column vector of $\mathbf{A}$. Your original understanding of matrix multiplication was pretty good.

Using the property of the transpose $\langle A^Tw,v\rangle = \langle w, Av\rangle$, I get:

$$\pmatrix{a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23}}\pmatrix{x \\ y \\ z} = \pmatrix{\langle A^T\pmatrix{1 \\ 0}, \pmatrix{x \\ y \\ z}\rangle \\ \langle A^T\pmatrix{0 \\ 1}, \pmatrix{x \\ y \\ z}\rangle} = \pmatrix{\langle \pmatrix{1 \\ 0}, A\pmatrix{x \\ y \\ z}\rangle \\ \langle \pmatrix{0 \\ 1}, A\pmatrix{x \\ y \\ z}\rangle} = \pmatrix{\operatorname{proj}_{e_1}(Ax) \\ \operatorname{proj}_{e_2}(Ax)}$$

That last bit should be absolutely clear -- The first component of $Ax$ is the projection of $Ax$ onto $e_1 = \pmatrix{1 \\ 0}$ and likewise for the second component.

• is there like a typo in your answer? Is $A$ suppose to be a row or the full matrix? – Pinocchio Nov 14 '17 at 16:52
• The question is what's the visual intuition in going from the first step to the second step. – 1110101001 Nov 20 '17 at 23:48

In terms of orthogonality, what it tells you is that the row space is the orthogonal complement to the Nul space. Thus every vector can be written uniquely as the sum of a vector in the row space of A and in the Nul space of A. This is a fundamental direct product relationship: $$\mathbb{R}^n = \mathrm{Row}\left(\mathbf{A}\right) \ \oplus \mathrm{Nul}\left(\mathbf{A}\right)$$

That is, if you let $\hat{\mathbf y}$ be the projection of $\mathbf{y}$ onto $\mathrm{Nul}\left(\mathbf{A}\right)$, then $\mathbf{y} - \mathbf{\hat{y}} \in \mathrm{Row}(\mathbf{A})$

To see this, let $$\mathbf{A} = \begin{bmatrix} \mathbf{a_1}^T \\ \mathbf{a_2}^T \\ \vdots \\ \mathbf{a_m}^T \end{bmatrix}$$

$\\$

Then

$$\mathbf{Ax} = \mathbf{A} = \begin{bmatrix} \mathbf{a_1}^T \\ \mathbf{a_2}^T \\ \vdots \\ \mathbf{a_m}^T \end{bmatrix} * \mathbf{x} = \begin{bmatrix} \mathbf{a_1}^T \mathbf{x} \\ \mathbf{a_2}^T \mathbf{x}\\ \vdots \\ \mathbf{a_m}^T \mathbf{x} \end{bmatrix}$$

Therefore $\mathbf{Ax} = \mathbf{0}$ if and only if $\mathbf{x}$ is orthogonal to each row of $\mathbf{A}$, and hence to the entire row space of A. Ironically, I have a book that refers to this and one other similar statement as the Fundamental Theorem of Linear Algebra. I don't think most would agree but it is an important result.

• I think the OP (or at least what I was curious about so I added the bounty) was for an intuitive/geometric explanation of why $$\left( \begin{array}{c} a_1^T \\ a_2^T \\ \end{array} \right).x=\left( \begin{array}{c} x a_1^T \\ x a_2^T \\ \end{array} \right)$$ in the first place, in the context of the visual understanding of dot products as projections and matrix-vector multiplication as a linear transformation – 1110101001 Nov 20 '17 at 23:24
• @1110101001 to answer the question in your comment right here, I think the reason they defined matrix multiplication this way is because of linear transformation the video is link in the question. How the definition of dot products arose and its meaning as product of rows is what still remains unknown to me (I might need to re-read the current answers). – Pinocchio Nov 20 '17 at 23:42
• @Pinocchio Yeah I know the reasoning for matrix multiplication as 3b1b presented in his video, as a linear combination of the columns. But I can't reconcile that intuition with the "alternative" method of doing matrix-vector multiplication by taking dot products of the rows with the vector. – 1110101001 Nov 20 '17 at 23:43
• @1110101001 I guess thats what my original question is about too :p if you do understand it, it would be awesome for u to share it :) – Pinocchio Nov 20 '17 at 23:45
• @DavidReed Right but the linear systems approach pretty much just views the matrix as a spreadsheet of numbers rather than as a linear transformation, which leaves out the elegance. It's clear that the dot products of the rows with the vector do work and are mathematically equivalent, but exactly why it works visually still remains a mystery – 1110101001 Nov 20 '17 at 23:46