Oh Times, $\otimes$ in linear algebra and tensors

Can I have some clarification of the different meanings of $\otimes$ as in the unifying and separating implications in basic linear algebra and tensors?

Here is some of the overloading of this symbol...

If $A$ is an $m \times n$ matrix and $B$ is a $p \times q$ matrix, then the Kronecker product A ⊗ B is the $mp \times nq$ block matrix:

$$A\color{red}{\otimes}B=\begin{bmatrix}a_{11}\mathbf B&\cdots&a_{1n}\mathbf B\\\vdots&\ddots&\vdots\\a_{m1}\mathbf B&\cdots&a_{mn}\mathbf B\end{bmatrix}$$

1.2. Outer product:

$\mathbf u \otimes \mathbf v = \mathbf{uv}^\top = \begin{bmatrix}u_1\\u_2\\u_3\\u_4\end{bmatrix}\begin{bmatrix}v_1&v_2&v_3&v_4\end{bmatrix}=\begin{bmatrix}u_1v_1&u_1v_2&u_1v_3\\u_2v_1&u_2v_2&u_2v_3\\u_3v_1&u_3v_2&u_3v_3\end{bmatrix}$

1. Definition of the tensor space:

\begin{align}T^p_q\,V &= \underset{p}{\underbrace{V\color{darkorange}{\otimes}\cdots\color{darkorange}{\otimes} V}} \color{darkorange}{\otimes} \underset{q}{\underbrace{V^*\color{darkorange}{\otimes}\cdots\color{darkorange}{\otimes} V^*}}:=\{T\, |\, T\, \text{ is a (p,q) tensor}\}\\[3ex]&=\{T: \underset{p}{\underbrace{V^*\times \cdots \times V^*}}\times \underset{q}{\underbrace{V\times \cdots \times V}} \overset{\sim}\rightarrow K\}\end{align}

1. Definition of the tensor product:

It takes $T\in T_q^p V$ and $S\in T^r_s V$ so that:

$$T\color{blue}{\otimes}S\in T_{q+s}^{p+r}V$$

defined as:

\begin{align}&(T\color{blue}{\otimes}S)(\underbrace{ \omega_1,\cdots,\omega_q,\cdots,\omega_{q+s}, v_1,\cdots,v_p,\cdots,v_{p+r}}_\text{'eats'})\\&:= T(\underbrace{\omega_1,\cdots,\omega_q, v_1,\cdots,v_p}_{\text{'eats up' p vec's + q covec's}\rightarrow \text{no.}})\underbrace{\cdot}_{\text{in the field}}S(\underbrace{\omega_{q+1},\cdots,\omega_{q+s}, v_{p+1},\cdots,v_{p+r}}_{\text{'eats up' p vec's and q covec's} \rightarrow\text{no.}})\end{align}

An example of, for instance, some operation like $\underbrace{e_{a_1}\color{blue}{\otimes}\cdots\color{blue}{\otimes}e_{a_p}\color{blue}{\otimes} \epsilon^{b_1}\color{blue}{\otimes}\cdots\color{blue}{\otimes}\epsilon^{b_q}}_{(p,q)\text{ tensor}}$ after settling for some basis could be helpful. For clarity this is a fragment of the more daunting expression:

$$T=\underbrace{\sum_{a_1=1}^{\text{dim v sp.}}\cdots\sum_{b_1=1}^{\text{dim v sp.}}}_{\text{p + q sums (usually omitted)}}\underbrace{\color{green}{T^{\overbrace{a_1,\cdots,a_p}^{\text{numbers}}}_{\quad\quad\quad\quad\underbrace{b_1,\cdots,b_q}_{\text{numbers}}}}}_{\text{a number}}\underbrace{\cdot}_{\text{S-multiplication}}\underbrace{e_{a_1}\color{blue}{\otimes}\cdots\color{blue}{\otimes}e_{a_p}\color{blue}{\otimes} \epsilon^{b_1}\color{blue}{\otimes}\cdots\color{blue}{\otimes}\epsilon^{b_q}}_{(p,q)\text{ tensor}}$$

showing how to recuperate a tensor from its components.

I realize that there is a connection as stated here:

The Kronecker product of matrices corresponds to the abstract tensor product of linear maps. Specifically, if the vector spaces $V, W, X$, and $Y$ have bases

$\{v_1, \cdots, v_m\}, \{w_1,\cdots, w_n\}, \{x_1,\cdots, x_d\},$ and $\{y_1, \cdots, y_e\}$, respectively,

and if the matrices $A$ and $B$ represent the linear transformations $S : V \rightarrow X$ and $T : W \rightarrow Y$, respectively in the appropriate bases, then the matrix $A ⊗ B$ represents

the tensor product of the two maps, $S ⊗ T : V ⊗ W → X ⊗ Y$ with respect to

the basis $\{v_1 ⊗ w_1, v_1 ⊗ w_2, \cdots, v_2 ⊗ w_1, \cdots, v_m ⊗ w_n\}$ of $V ⊗ W$ and the similarly defined basis of $X ⊗ Y$ with the

property that $A ⊗ B(v_i ⊗ w_j) = (Av_i) ⊗ (Bw_j)$, where $i$ and $j$ are integers in the proper range.

But it is still elusive...

• They are all the same. Taking tensor products is a functor, meaning that it operates on both vector spaces and on linear maps in a compatible way. 2 is an example of tensor product operating on vector spaces, 1 is an example of it operating on linear maps, and 3 is secretly an example of it operating on linear maps (vectors in a vector space correspond to linear maps from a one-dimensional vector space to it). – Qiaochu Yuan Feb 24 '17 at 1:11
• Why not start with "Let $\otimes$ be..."? Defining well what symbols mean is the first and most important step to write a clear answer! – Daniel Robert-Nicoud Feb 24 '17 at 1:11
• This kind of overloading is extremely common everywhere and without it everything would be terrible. You use the same symbol $+$ to denote addition of natural numbers, integers, real numbers, complex numbers, matrices, functions, etc. and this is all fine and good. If someone asked you to use a different symbol for each of these you would punch them. – Qiaochu Yuan Feb 24 '17 at 1:12
• @QiaochuYuan I hope you write a formal answer, but just in case... What would immediately come to mind if the symbol $\otimes$ was shown to a completely drunk and incoherent mathematician with a PhD in differential geometry? Would it be the operation in (1)? – Antoni Parellada Feb 24 '17 at 1:25
• No, it would probably be either 2 or 3. – Qiaochu Yuan Feb 24 '17 at 1:34

If $V$ and $W$ are vector spaces, you can form a third vector space from them called their tensor product $V \otimes W$. The tensor product consists of sums of certain vectors called "pure tensors," which are written $v \otimes w$ where $v \in V, w \in W$, subject to certain rules, e.g. $(v_1 + v_2) \otimes w = v_1 \otimes w + v_2 \otimes w$. For a complete list of these rules see Wikipedia. In practice you'll do fine if you remember the following:

If $v_1, \dots v_n$ is a basis of $V$ and $w_1, \dots w_m$ is a basis of $W$, then the pure tensors $v_i \otimes w_j, 1 \le i \le n, 1 \le j \le m$ form a basis of $V \otimes W$. In particular, $\dim V \otimes W = \dim V \times \dim W$.

If $T : V_1 \to V_2$ and $S : W_1 \to W_2$ are two linear maps, you can form a third linear map from them which is also called their tensor product

$$T \otimes S : V_1 \otimes W_1 \to V_2 \otimes W_2.$$

It is completely determined by how it behaves on pure tensors, which is

$$(T \otimes S)(v \otimes w) = T(v) \otimes S(w).$$

The relationship between these two uses of the term "tensor product" is given formally by the notion of a functor.

Tensor product notation for linear maps is compatible with the notation $v \otimes w$ for pure tensors in the following sense. A vector $v \in V$ in a vector space is the same thing as a linear map $v : 1 \to V$ from the one-dimensional vector space $1$ given by the underlying field to $V$, and if $v : 1 \to V$ and $w : 1 \to W$ are two vectors in $V, W$, then their tensor product as linear maps $v \otimes w : 1 \otimes 1 \to V \otimes W$ corresponds to the pure tensor $v \otimes w$, where we use that there's a canonical isomorphism $1 \otimes 1 \cong 1$.

The Kronecker product is a description of the tensor product of linear maps with respect to a choice of basis for all of the vector spaces involved. Formally, with notation as above, if

1. $B_1, B_2$ are bases for $V_1, V_2$,
2. $C_1, C_2$ are bases for $W_1, W_2$,
3. given bases $B_i, C_i$ of $V_i, W_i$, we write $B_i \otimes C_i$ for the corresponding basis of $V_i \otimes W_i$ as in the highlighted area above, and
4. we write $_{B_2}[T]_{B_1}$ to refer to the matrix of a linear transformation $T : V_1 \to V_2$ with respect to a basis $B_1$ of $V_1$ and a basis $B_2$ of $V_2$,

then we have

$$_{B_2 \otimes C_2}[T \otimes S]_{B_1 \otimes C_1} = \, _{B_2}[T]_{B_1} \otimes \, _{C_2}[S]_{C_1}$$

where on the LHS $\otimes$ means the tensor product of linear maps and on the RHS $\otimes$ means the Kronecker product.

One final remark: the definition of spaces of tensors you give in 2) is a terrible definition that I've only seen in some textbooks on differential geometry. It is absolutely the wrong way to think about tensors.

• I am sympathetic to your disdain for the differential geometer's perspective. That being said, I find it helpful for both sanity and understanding to note that the space of multilinear forms is canonically isomorphic to the dual of the tensor product of vector spaces. Moreover, the existence of such a canonical isomorphism is precisely the content of the universal property by which tensor products are defined. – Omnomnomnom Feb 24 '17 at 2:07
• I'm curious, because I don't really get what you mean by the last paragraph. The thing he cites in 2) contains several expressions. Which part do you object to and how would you phrase it better? Are you referring to expressing Hom-sets as tensor products? – tomasz Feb 24 '17 at 14:41
• @tomasz: it's the last bit about expressing tensor products in terms of multilinear functions. The goal here is to avoid having to talk abstractly about pure tensors and relations, but it comes at the cost of being unnecessarily convoluted - you already have to be comfortable with taking double duals to convince yourself that this definition does the right thing. – Qiaochu Yuan Feb 24 '17 at 19:31
• @QiaochuYuan: and what is the "right thing" you have in mind? I am not a geometer, but I've mostly learned about tensor products in presence of geometers, so my perspective may be a little skewed here, but it seems to me like that (multilinearity and the universal property) is precisely what the definition is about, while the pure tensors and relations only give a construction of the universal object. Sure, they are useful to understand what's going on, but that is not what the tensor product is, just like the completion of a metric space is not the set of classes of Cauchy sequences. – tomasz Feb 24 '17 at 20:06
• @QiaochuYuan: I admit, I have never given much thought to infinite dimensional tensor spaces. Which two constructions you are saying are different? Sorry if I'm bothering you, but now you got me real curious. :-) And I wouldn't mind being able to contradict a geometer. ;-) – tomasz Feb 24 '17 at 20:17