# Geometric intuition of tensor product

Let $V$ and $W$ be two algebraic structures, $v\in V$, $w\in W$ be two arbitrary elements.

Then, what is the geometric intuition of $v\otimes w$, and more complex $V\otimes W$ ? Please explain for me in the most concrete way (for example, $v, w$ are two vectors in 2 dimensional vector spaces $V, W$)

Thanks

• – Willie Wong Mar 2 '12 at 10:30
• @WillieWong : Thank you very much. Yes I have. However, the links above just mentioned the algebraic properties/structures. What I could not imagine is the concrete picture of tensor product. – Arsenaler Mar 2 '12 at 13:29
• The tensor product is an algebraic convenience and does not in general admit a geometric interpretation. (On the other hand, it is easy to give a geometric interpretation of the wedge product.) – Zhen Lin Mar 2 '12 at 17:18
• Do you have a geometric appreciation for bilinear forms? – Dylan Moreland Apr 1 '12 at 16:57
• @eduardo: Related: math.stackexchange.com/questions/309838/… – user23238 Feb 23 '13 at 9:04

You want to stay concrete, so let's let $V$ be a two-dimensional real vector space and $W = \operatorname{Hom}(V,\mathbb{R})$. Then $V = T^1_0(V)$ and $W = T^0_1(V)$, so for any $v\in V$ and $w\in W$, $v\otimes w\in T^1_1(V)$.

Each of $v,w$ has two components; $v = v^1e_1 + v^2e_2$ and $w = w_1e^1 + w_2e^2$. Here $e_1,e_2$ is a basis for $V$ and $e^1,e^2$ is the dual basis in $V^*$.

The components of $v\otimes w$ are all the components of $v$ times all the components of $w$: $$(v\otimes w)^i_j = v^iw_j.$$

To see this, observe that $(v\otimes w)(\theta, x)=v(\theta)w(x),$ so that $(v\otimes w)(e^i, e_j) = v(e^i)w(e_j).$

More generally, if $A = A^{i_1\cdots i_p}_{j_1\cdots j_q}\in T^p_q$ and $B = B^{k_1\cdots k_r}_{l_1\cdots l_s}\in T^r_s$, then

$$(A\otimes B)^{i_1\cdots i_pk_1\cdots k_r}_{j_1\cdots j_q l_1\cdots l_s} = A^{i_1\cdots i_p}_{j_1\cdots j_q}B^{k_1\cdots k_r}_{l_1\cdots l_s}.$$

Note that our example is from the derived tensor algebra over a two-dimensional vector space, $T^p_q(V) = (V^*)^{\otimes p}V^{\otimes q}.$ Hopefully this helps build your intuition about the case where $V$ and $W$ are two vector spaces of potentially different dimension.

• Double multi-indices in a question regarding "geometric intuition"! – Martin Brandenburg Nov 25 '15 at 9:21
• “$T^p_q(V) = (V^*)^{\otimes p}V^{\otimes q}$” – shouldn’t that be $T^p_q(V) = V^{\otimes p} \otimes (V^*)^{\otimes q}$? – Socob May 3 '18 at 3:30

The way I like to think about this is that if we have two vector spaces $V$ and $W$, their tensor product $V \otimes W$ intuitively is the idea "for each vector $v\in V$, attach to it the entire vector space $W$." Notice however that since we have $V\otimes W \cong W \otimes V$ we can rephrase this as the idea "for every $w \in W$ attach to it the entire vector space $V$." What distinguishes $V\times W$ from $V\otimes W$ is that the tensor space is essentially a linearized version of $V\times W$. To obtain $V\otimes W$ we start with the product space $V\times W$ and consider the free abelian group over it, denoted by $\mathcal{F}(V\times W)$. Given this larger space, that admits scalar products of ordered pairs, we set up equivalence relations of the form $(av,w) \sim a(v,w) \sim (v, aw)$ with $v\in V, w \in W$ and $a \in \mathbb{F}$ (a field), so that we identify points in the product space that yield multilinear relationships in the quotient space generated by these relationships. If $\mathcal{S}$ is the subspace that is spanned by these equivalence relations, we define $V\otimes W = \mathcal{F}(V\times W)/\mathcal{S}$.

Personally, I like to understand the tensor product in terms of multilinear maps and differential forms since this further makes the notion of tensor product more intuitive for me (and this is typically why tensor products are used in physics/applied math). For instance if we take an $n$-dimensional real vector space $V$, we can consider the collection of multilinear maps of the form

$$F: \underbrace{V \otimes \ldots \otimes V}_{k \; times} \to \mathbb{R}$$

where $F \in V^* \otimes \ldots \otimes V^*$, and this tensor space has basis vectors $dx^{i_1} \otimes \ldots \otimes dx^{i_k}$ and $\{i_1, \ldots, i_k\} \subseteq \{1, \ldots, n\}$. Notice now that given $k$ vectors $v_1, \ldots, v_k \in V$ we have that multilinear tensor functional $dx^{i_1} \otimes \ldots \otimes dx^{i_k}$ acts on the ordered pair $(v_1, \ldots, v_k)$ as

$$dx^{i_1} \otimes \ldots \otimes dx^{i_k}(v_1, \ldots, v_k) \;\; =\;\; dx^{i_1}(v_1) \ldots dx^{i_k}(v_k).$$

Where we have that $dx^{i_j}(v_j)$ takes the vector $v_j$ and picks out its $i_j$-th component. Extending this notion to differential forms where instead we would have basis elements $dx^{i_1} \wedge \ldots \wedge dx^{i_k}$ (where antisymmetrization is taken into account), we would have that this basis form would take a set of $k$ vectors and in some sense "measure" how much these vectors overlap with the subspace generated by the basis $\{x_{i_1}, \ldots, x_{i_k}\}$.

• Concerning the first part of your answer, I have doubts on your idea on constructing $V \otimes W$ out of $V \times W$ via an equivalence realtion. You are forgetting that elements of $V \otimes W$ do not have to have the form $v \otimes w$ for $v \in V$ and $w \in W$. Also, the dimension of $V \otimes W$ (product of dimensions) is often larger than the dimension of $V \times W$ (sum of dimensions). – Matthias Klupsch Dec 14 '16 at 11:42
• @MatthiasKlupsch Thank you for pointing this out. What I missed is that instead of looking simply at $V\times W$, we look at the free abelian group over $V\times W$. We then mod out by relations on this space. I'll edit my answer to reflect this. – Mnifldz Dec 14 '16 at 20:37
• I believe your equivalence relation is still incomplete, you forgot relations like $(v, w) + ( v , t) - (v , (w + t)) = 0$. Also, you have to consider the relations which you want to have first, these do not give you an equivalence relation directly, but the subspace generated by the relations induces an equivalence relation (two elements being equivalent iff they lie in the same coset). – Matthias Klupsch Dec 15 '16 at 16:00
• @MatthiasKlupsch My answer was meant as a brief qualitative description, especially since the OP was asking about geometric intuition behind tensor spaces and not so much a detailed construction. If you're not satisfied with what's written here, there's a whole internet of resources for you to find what you're looking for. – Mnifldz Dec 15 '16 at 16:52
• I see. I did not think you had shortened the 'definition' of tensor product deliberately but assumed you had forgotten that this part existed, so I wanted to point it out to you to improve the quality of the answer. I understand I was wrong with my assumption. – Matthias Klupsch Dec 15 '16 at 20:39

The difference between the ordered pair $(v,w)$ of vectors and the tensor product $v\otimes w$ of vectors is that for a scalar $c\not\in\{0,1\}$, the pair $(cv,\;w/c)$ is different from the pair $(v,w)$, but the tensor product $(cv)\otimes(w/c)$ is the same as the tensor product $v\otimes w$.

geometric intuition of $$v\otimes w$$ [...] in the most concrete way

Let me try to throw in an answer on freshmen level. I deliberately try to leave out any information that is not essential to the problem, e.g. don't bother about the specific dimensions of the quantities.

Say we have a matrix $$V$$ and a vector $$w$$ resulting in $$Vw=v$$ (implying consistent dimensions of the matrix $$V$$ and of the column vectors $$v$$ and $$w$$ over, e.g., $$\mathbb{R}$$). Now, let's forget about $$V$$ and try to reconstruct it given only $$v$$ and $$w$$.

It is generally not possible to fully reconstruct it from this little information, but we can do our best by trying $$\tilde{V}:=v\otimes w \frac{1}{\lVert w\rVert^2}$$ In this case the tensor product can be rewritten to $$v\otimes w := v\,w^{\sf T}$$ Then, checking the ability of $$\tilde{V}$$ to approximate the action of $$V$$ we see $$\tilde{V}w=v\,w^{\sf T}w\frac{1}{\lVert w\rVert^2}=v$$ which is the best we can do.

To make it more concrete, let's throw in some numbers in the easiest non-trivial case: $$Vw=\left(\begin{matrix}1&0\\0&2\end{matrix}\right)\left(\begin{matrix}1\\0\end{matrix}\right)=\left(\begin{matrix}1\\0\end{matrix}\right)=v$$ Then $$\tilde{V}=\left(\begin{matrix}1\\0\end{matrix}\right)\left(\begin{matrix}1&0\end{matrix}\right)=\left(\begin{matrix}1&0\\0&0\end{matrix}\right)\approx V=\left(\begin{matrix}1&0\\0&2\end{matrix}\right)$$ The subspace spanned by $$w=\left(\begin{matrix}1\\0\end{matrix}\right)$$ is accurately mapped by $$\tilde{V}$$.

To give more (very elementary) geometric insight, let's make the matrix slightly more interesting: $$V=\left(\begin{matrix}1&1\\0&2\end{matrix}\right)$$ This leaves the $$\left(\begin{matrix}1\\0\end{matrix}\right)$$-space invariant and scales and shears the $$\left(\begin{matrix}0\\1\end{matrix}\right)$$-space. We now "test" the matrix by the "arbitrary" vector $$w=\left(\begin{matrix}1\\1\end{matrix}\right)$$ and get $$Vw=\left(\begin{matrix}1&1\\0&2\end{matrix}\right)\left(\begin{matrix}1\\1\end{matrix}\right)=\left(\begin{matrix}2\\2\end{matrix}\right)=v$$ and the approximate reconstruction $$\tilde{V}=\left(\begin{matrix}2\\2\end{matrix}\right)\left(\begin{matrix}1&1\end{matrix}\right)\frac{1}{2}=\left(\begin{matrix}1&1\\1&1\end{matrix}\right)\approx V=\left(\begin{matrix}1&1\\0&2\end{matrix}\right)$$ We have lost the information about the invariance of the $$\left(\begin{matrix}1\\0\end{matrix}\right)$$-axis and the scaling & shearing behavior of the $$\left(\begin{matrix}0\\1\end{matrix}\right)$$-axis, but the diagonal $$\left(\begin{matrix}1\\1\end{matrix}\right)$$-space's behavior is captured by $$\tilde{V}$$.

To increase the level of this answer by an $$\varepsilon>0$$, the spectral decomposition of a diagonalizable matrix $$V$$ should be mentioned. If $$v_1,\dots,v_n$$ are normalized eigenvectors with corresponding eigenvalues $$\lambda_1,\dots,\lambda_n$$, then $$V=\sum_{i=1}^n v_i\otimes v_i \lambda_i$$

More intuitively, we can say - very roughly speaking - that the tensor product $$v\otimes w$$ is the pseudo-inverse operation of the matrix vector multiplication $$Vw=v$$.

To finish this, I'd like to give one concrete (and special) application of the tensor product in partial differential equations. It links this question to the divergence theorem, which is itself very intuitive. To keep it descriptive, let's stick to the case of solid mechanics (because everyone has inherent understanding of bodies and forces), although this holds for all elliptic PDEs in general.

Say we have an elastic body (e.g. a rubber band) occupying a domain $$\Omega\subset\mathbb{R}^3$$. Some part of it is subject to Dirichlet boundary conditions (e.g. you're statically stretching it). This results internal stresses, which are typically represented by matrices $$\sigma\in\mathbb{R}^{3\times3}$$. The system of PDEs describing this state is $${\rm div}(\sigma)=0$$ (considering the boundary conditions and the governing material law which closes these equations).

In some applications you are interested in the average stress throughout the body $$\Omega$$ $$\bar{\sigma}=\frac{1}{\mu(\Omega)}\int_\Omega \sigma \, {\rm d}\mu$$ The divergence theorem implies that the average stress can be represented by the surface integral $$\bar{\sigma}=\frac{1}{\mu(\Omega)}\int_{\partial\Omega} \left(\sigma n\right) \otimes x \, {\rm d}A$$ where $$\partial\Omega$$ is the boundary of $$\Omega$$, $$n$$ is the surface normal, and $$x$$ the coordinate vector.

This means that the volume average of a matrix can be represented integrating the pseudo-inverse of its application over the surface.