Intuitive explanation of outer product

Question

The inner product between two vectors is the product of length of first vector and the length of projection of second vector on to the first vector.

When I take an outer product its result is a matrix. I understand how to calculate it but I am not able to find out what it represents intuitively and why would it be useful. I have searched about it but have not found some simple explanation of it for myself.

So any easy to understand explanation of it would be much appreciated.

Many thanks!

Answer 1

Here is a concrete interpretation: outer products are the abstract version of matrices with a single nonzero entry. Such special matrices span the space of of all matrices (of a fixed size). Does that convince you that this concept of outer product should be useful?

First let's recall the nonintuitive definition of the outer product. For vectors $\mathbf u$ in $\mathbf R^m$ and $\mathbf v$ in $\mathbf R^n$, their outer product is $\mathbf u \mathbf v^\top$. This is an $m \times n$ matrix: if $\mathbf u = (a_1, \ldots, a_m)$ and $\mathbf v = (b_1,\ldots,b_n)$ are viewed as column vectors then $\mathbf u \mathbf v^\top = (a_ib_j)$: the product of an $m \times 1$ and $1 \times n$ matrix in that order is an $m \times n$ matrix. This doesn't explain what the matrix $\mathbf u \mathbf v^\top$ means, and that's your question.

Abstractly, an $m \times n$ matrix is a linear map $\mathbf R^n \to \mathbf R^m$. Is there something special about linear maps having a matrix representation of the form $\mathbf u \mathbf v^\top$? Yes! Most matrices can't be described in that way. To figure out what makes matrices $\mathbf u \mathbf v^\top$ special, let's see what their effect is on each $\mathbf x$ in $\mathbf R^n$. The linear map with matrix representation $\mathbf u \mathbf v^\top$ has the effect $$ \mathbf x \mapsto (\mathbf u \mathbf v^\top)\mathbf x = \mathbf u (\mathbf v^\top\mathbf x) = \mathbf u (\mathbf v \cdot \mathbf x) = (\mathbf v \cdot \mathbf x)\mathbf u. $$ Notice the value of this map is a scalar multiple of $\mathbf u$ no matter what $\mathbf x$ is. So this linear map $\mathbf R^n \to \mathbf R^m$ has a $1$-dimensional image (the scalar multiples of $\mathbf u$) except in the case that $\mathbf u$ or $\mathbf v$ is $\mathbf 0$.

For example, if $u = \binom{a}{b}$ and $v = \binom{c}{d}$ then $\mathbf u \mathbf v^\top = \binom{a}{b}(c \ d) = \left(\begin{smallmatrix}ac&ad\\bc&bd \end{smallmatrix}\right)$. For $\binom{x}{y} \in \mathbf R^2$, $\left(\begin{smallmatrix}ac&ad\\bc&bd \end{smallmatrix}\right)\binom{x}{y} = \binom{acx+ady}{bcx+bdy} = (cx+dy)\binom{a}{b}$, which is a scalar multiple of $\mathbf u$ no matter what $\mathbf x$ is, and the scalar used is $cx + dy = \mathbf v \cdot \mathbf x$.

Claim: every linear map $L \colon \mathbf R^n \to \mathbf R^m$ with a $1$-dimensional image has the form $L(\mathbf x) = (\mathbf v \cdot \mathbf x)\mathbf u$ for some nonzero $\mathbf u$ in $\mathbf R^m$ and $\mathbf v$ in $\mathbf R^n$.

I am not saying linear maps with a $1$-dimensional image look like $\mathbf x \mapsto (\mathbf v \cdot \mathbf x)\mathbf u$ for unique $\mathbf u$ and $\mathbf v$, since for nonzero scalars $c$, $(\mathbf v \cdot \mathbf x)\mathbf u = (c\mathbf v \cdot \mathbf x)((1/c)\mathbf u)$. That is, $(1/c)\mathbf u$ and $c \mathbf v$ define the same linear map $\mathbf R^n \to \mathbf R^m$ for all nonzero scalars $c$.

Proof: Let $\mathbf u$ in $\mathbf R^n$ be an arbitrary nonzero vector in the image of $L$. Since the image is $1$-dimensional, $L(\mathbf R^m) = \mathbf R\mathbf u$. So for each $\mathbf x$ in $\mathbf R^m$, $L(\mathbf x) = \varphi(\mathbf x)\mathbf u$ for a unique real number $\varphi(\mathbf x)$. Linearity of $L$ implies $\varphi \colon \mathbf R^m \to \mathbf R$ is linear and not identically $0$ (otherwise $L$ would have image $\{\mathbf 0\}$). And here is the key point: every nonzero linear map $\mathbf R^m \to \mathbf R$ is forming an inner product with some nonzero vector in $\mathbf R^m$. So there is a nonzero $\mathbf v$ in $\mathbf R^m$ such that $\varphi(\mathbf x) = \mathbf x \cdot \mathbf v$ for all $\mathbf x \in \mathbf R^m$. Therefore $$ L(\mathbf x) = \varphi(\mathbf x)\mathbf u = (\mathbf x \cdot \mathbf v)\mathbf u = \mathbf u (\mathbf v \cdot \mathbf x) = \mathbf u (\mathbf v^\top \mathbf x) = (\mathbf u \mathbf v^\top) \mathbf x, $$ so as an $m \times n$ matrix, $L$ is the outer product $\mathbf u \mathbf v^\top$.

Thus we have a conceptual explanation of an outer product of a nonzero vector in $\mathbf R^m$ and a nonzero vector in $\mathbf R^n$ (in that order): it is the same thing as a linear map $\mathbf R^m \to \mathbf R^n$ with a $1$-dimensional image. In order to allow one of the vectors involved to be $\mathbf 0$, we can say an outer product of a vector in $\mathbf R^m$ and a vector in $\mathbf R^n$ (in that order) is the same thing as a linear map $\mathbf R^m \to \mathbf R^n$ whose image has dimension at most $1$.

The conceptual meaning of outer products is best revealed using the language of tensor products (if you know what those are). The space of all linear maps $\mathbf R^n \to \mathbf R^m$ is $\mathbf R^m \otimes_{\mathbf R} (\mathbf R^n)^*$, where $(\mathbf R^n)^*$ is the dual space of $\mathbf R^n$. The outer product $\mathbf u \mathbf v^\top$ corresponds to the simple tensor $\mathbf u \otimes \varphi_{\mathbf v}$, where $\varphi_{\mathbf v}$ is the linear map "form the inner product with $\mathbf v$ on $\mathbf R^n$". So the fact that $(1/c)\mathbf u$ and $c \mathbf v$ for nonzero scalars $c$ define the same linear map $\mathbf R^n \to \mathbf R^m$ corresponds to the tensor property $\mathbf u \otimes \varphi_{\mathbf v} = (1/c)\mathbf u \otimes c\varphi_{\mathbf v}$. Every tensor is a sum of at most $mn$ simple tensors, which corresponds (using the standard bases of $\mathbf R^m$ and $\mathbf R^n$) to saying every $m \times n$ matrix is a sum of matrices with a single nonzero entry in the matrix.

Answer 2

Right, as you say, the outer product of two vectors looks like a matrix. As such, we should think of what happens when it acts on a vector. What we get out is the left vector in the outer product, scaled by the inner product of the right vector and the new vector. This does have quite a few uses, here's one from quantum mechanics, using bra ket notation (and implying summation by repeated indices)

Let $|\psi \rangle$ be a vector. As such it's an abstract object without a coordinate representation. Now let an orthonormal basis for the space in which this vector lies be

$$\{ | e_i \rangle \}$$.

Importantly, the matrix (again, with a repeated sum implied,

$$| e_i \rangle \langle e_i |$$

is the identity matrix. Then, we can apply it freely to the vector of interest:

$$| e_i \rangle \langle e_i | \psi \rangle$$

The inner product on the right is a complex number, and the vector on the left is an element of our orthonormal basis. Then, we could represent our vector by listing these complex scalars in order, with the understanding that they multiply our basis vectors.

If we decide we don't like our basis vectors, we could choose a new orthonormal basis and change our vector's representation by multiplying by this new resolution of the identity, with its new summed outer product. We would then obtain an object like

$$ | d_j \rangle \langle d_j | e_i \rangle \langle e_i|$$

which now has an outer product between two distinct sets of vectors (albeit with a complex number from the inner product rescaling each one). This could be represented as a matrix with the $i,j$ element given by this inner product, so that the matrix itself is a weighted sum of outer products.

Or here's a simpler example. Let's say we want to rotate a real vector in the plane by a quarter-turn. Then, we should take $(x,y) \rightarrow (y,-x)$, right? One way to write the matrix/linear operator that does this is as

$$|x\rangle \langle y |-|y\rangle \langle x |$$.