Oh Times, $\otimes$ in linear algebra and tensors

Question

Can I have some clarification of the different meanings of $\otimes$ as in the unifying and separating implications in basic linear algebra and tensors?

Here is some of the overloading of this symbol...

1.1. Kronecker matrix product:

If $A$ is an $m \times n$ matrix and $B$ is a $p \times q$ matrix, then the Kronecker product A ⊗ B is the $mp \times nq$ block matrix:

$$A\color{red}{\otimes}B=\begin{bmatrix}a_{11}\mathbf B&\cdots&a_{1n}\mathbf B\\\vdots&\ddots&\vdots\\a_{m1}\mathbf B&\cdots&a_{mn}\mathbf B\end{bmatrix}$$

---

1.2. Outer product:

$\mathbf u \otimes \mathbf v = \mathbf{uv}^\top = \begin{bmatrix}u_1\\u_2\\u_3\\u_4\end{bmatrix}\begin{bmatrix}v_1&v_2&v_3&v_4\end{bmatrix}=\begin{bmatrix}u_1v_1&u_1v_2&u_1v_3\\u_2v_1&u_2v_2&u_2v_3\\u_3v_1&u_3v_2&u_3v_3\end{bmatrix}$

---

2. Definition of the tensor space:

$$\begin{align}T^p_q\,V &= \underset{p}{\underbrace{V\color{darkorange}{\otimes}\cdots\color{darkorange}{\otimes} V}} \color{darkorange}{\otimes} \underset{q}{\underbrace{V^*\color{darkorange}{\otimes}\cdots\color{darkorange}{\otimes} V^*}}:=\{T\, |\, T\, \text{ is a (p,q) tensor}\}\\[3ex]&=\{T: \underset{p}{\underbrace{V^*\times \cdots \times V^*}}\times \underset{q}{\underbrace{V\times \cdots \times V}} \overset{\sim}\rightarrow K\}\end{align}$$

---

3. Definition of the tensor product:

It takes $T\in T_q^p V$ and $S\in T^r_s V$ so that:

$$T\color{blue}{\otimes}S\in T_{q+s}^{p+r}V$$

defined as:

$$\begin{align}&(T\color{blue}{\otimes}S)(\underbrace{ \omega_1,\cdots,\omega_q,\cdots,\omega_{q+s}, v_1,\cdots,v_p,\cdots,v_{p+r}}_\text{'eats'})\\&:= T(\underbrace{\omega_1,\cdots,\omega_q, v_1,\cdots,v_p}_{\text{'eats up' p vec's + q covec's}\rightarrow \text{no.}})\underbrace{\cdot}_{\text{in the field}}S(\underbrace{\omega_{q+1},\cdots,\omega_{q+s}, v_{p+1},\cdots,v_{p+r}}_{\text{'eats up' p vec's and q covec's} \rightarrow\text{no.}})\end{align}$$

---

An example of, for instance, some operation like $\underbrace{e_{a_1}\color{blue}{\otimes}\cdots\color{blue}{\otimes}e_{a_p}\color{blue}{\otimes} \epsilon^{b_1}\color{blue}{\otimes}\cdots\color{blue}{\otimes}\epsilon^{b_q}}_{(p,q)\text{ tensor}}$ after settling for some basis could be helpful. For clarity this is a fragment of the more daunting expression:

$$ T=\underbrace{\sum_{a_1=1}^{\text{dim v sp.}}\cdots\sum_{b_1=1}^{\text{dim v sp.}}}_{\text{p + q sums (usually omitted)}}\underbrace{\color{green}{T^{\overbrace{a_1,\cdots,a_p}^{\text{numbers}}}_{\quad\quad\quad\quad\underbrace{b_1,\cdots,b_q}_{\text{numbers}}}}}_{\text{a number}}\underbrace{\cdot}_{\text{S-multiplication}}\underbrace{e_{a_1}\color{blue}{\otimes}\cdots\color{blue}{\otimes}e_{a_p}\color{blue}{\otimes} \epsilon^{b_1}\color{blue}{\otimes}\cdots\color{blue}{\otimes}\epsilon^{b_q}}_{(p,q)\text{ tensor}}$$

showing how to recuperate a tensor from its components.

---

I realize that there is a connection as stated here:

The Kronecker product of matrices corresponds to the abstract tensor product of linear maps. Specifically, if the vector spaces $V, W, X$, and $Y$ have bases

$\{v_1, \cdots, v_m\}, \{w_1,\cdots, w_n\}, \{x_1,\cdots, x_d\},$ and $\{y_1, \cdots, y_e\}$, respectively,

and if the matrices $A$ and $B$ represent the linear transformations $S : V \rightarrow X$ and $T : W \rightarrow Y$, respectively in the appropriate bases, then the matrix $A ⊗ B$ represents

the tensor product of the two maps, $S ⊗ T : V ⊗ W → X ⊗ Y$ with respect to

the basis $\{v_1 ⊗ w_1, v_1 ⊗ w_2, \cdots, v_2 ⊗ w_1, \cdots, v_m ⊗ w_n\}$ of $V ⊗ W$ and the similarly defined basis of $X ⊗ Y$ with the

property that $A ⊗ B(v_i ⊗ w_j) = (Av_i) ⊗ (Bw_j)$, where $i$ and $j$ are integers in the proper range.

But it is still elusive...

Answer 1

If $V$ and $W$ are vector spaces, you can form a third vector space from them called their tensor product $V \otimes W$. The tensor product consists of sums of certain vectors called "pure tensors," which are written $v \otimes w$ where $v \in V, w \in W$, subject to certain rules, e.g. $(v_1 + v_2) \otimes w = v_1 \otimes w + v_2 \otimes w$. For a complete list of these rules see Wikipedia. In practice you'll do fine if you remember the following:

If $v_1, \dots v_n$ is a basis of $V$ and $w_1, \dots w_m$ is a basis of $W$, then the pure tensors $v_i \otimes w_j, 1 \le i \le n, 1 \le j \le m$ form a basis of $V \otimes W$. In particular, $\dim V \otimes W = \dim V \times \dim W$.

If $T : V_1 \to V_2$ and $S : W_1 \to W_2$ are two linear maps, you can form a third linear map from them which is also called their tensor product

$$T \otimes S : V_1 \otimes W_1 \to V_2 \otimes W_2.$$

It is completely determined by how it behaves on pure tensors, which is

$$(T \otimes S)(v \otimes w) = T(v) \otimes S(w).$$

The relationship between these two uses of the term "tensor product" is given formally by the notion of a functor.

Tensor product notation for linear maps is compatible with the notation $v \otimes w$ for pure tensors in the following sense. A vector $v \in V$ in a vector space is the same thing as a linear map $v : 1 \to V$ from the one-dimensional vector space $1$ given by the underlying field to $V$, and if $v : 1 \to V$ and $w : 1 \to W$ are two vectors in $V, W$, then their tensor product as linear maps $v \otimes w : 1 \otimes 1 \to V \otimes W$ corresponds to the pure tensor $v \otimes w$, where we use that there's a canonical isomorphism $1 \otimes 1 \cong 1$.

The Kronecker product is a description of the tensor product of linear maps with respect to a choice of basis for all of the vector spaces involved. Formally, with notation as above, if

1. $B_1, B_2$ are bases for $V_1, V_2$,
2. $C_1, C_2$ are bases for $W_1, W_2$,
3. given bases $B_i, C_i$ of $V_i, W_i$, we write $B_i \otimes C_i$ for the corresponding basis of $V_i \otimes W_i$ as in the highlighted area above, and
4. we write $_{B_2}[T]_{B_1}$ to refer to the matrix of a linear transformation $T : V_1 \to V_2$ with respect to a basis $B_1$ of $V_1$ and a basis $B_2$ of $V_2$,

then we have

$$_{B_2 \otimes C_2}[T \otimes S]_{B_1 \otimes C_1} = \, _{B_2}[T]_{B_1} \otimes \, _{C_2}[S]_{C_1}$$

where on the LHS $\otimes$ means the tensor product of linear maps and on the RHS $\otimes$ means the Kronecker product.

One final remark: the definition of spaces of tensors you give in 2) is a terrible definition that I've only seen in some textbooks on differential geometry. It is absolutely the wrong way to think about tensors.