I am trying to understand the concept of tensors. I seem to understand that they are generalization of vectors: They are subject to similar basis transformations with vectors but I am somewhat confused about their relations with matrices. In 3 dimensional Euclidian space, a rank 1 tensor is represented with a vector, a rank 2 tensor is represented with a 3x3 matrix, a rank 3 tensor is a 3x3x3 sized three dimensional array and so on. Assuming we are in 3 dimensional space, for example, is it possible to have an arbitrary sized matrix; say 6x7 sized matrix as the representation of a tensor? Or can they be only represented with arrays with dimensions of three's powers? I think that this is not possible since an arbitrary sized matrix cannot undergo basis transformations.

My second question is about tensor multiplications. This operation seems like a cartesian product between the elements of two tensors, for example the tensor product of a rank 1 (3 elements) and rank 2 (9 elements) tensor generates a rank 3 three tensor (27 elements). Is this operation related to standard matrix multiplication somehow? For example, we can do matrix multiplication between 3x1 and 1x3 vectors and obtain a 3x3 matrix. Isn't this a tensor product between two rank 1 tensors in the same time? Is this a coincidence?

I think putting tensors in the right context will clear much of this up. I'll stick with $\mathbb R^3$ since that's the example you use. Rank $2$ tensors are elements of the so-called tensor product of $\mathbb R^3$ with itself, which is denoted by $\mathbb R^3 \otimes \mathbb R^3$. This space consists of all linear combinations of expressions of the form $u \otimes v$ under the stipulations that: $$u\otimes(v + w) = u\otimes v + u\otimes w,$$ $$(u+v)\otimes w = u\otimes w + v\otimes w, \text{and}$$ $$u\otimes (cv) = c(u \otimes v) = (cu) \otimes v$$ where $u,v,w$ are vectors and $c,d$ are scalars.
Taking the standard basis $e_1,e_2,e_3$ of $\mathbb R^3$, any rank $2$ tensor can then be written as a linear combination of the $9$ "pure" tensors $e_i \otimes e_j$ for $i,j = 1,2,3$. The $9$ scalars you take as coefficients in such a linear combination make up the $3 \times 3$ matrix which "represents" that rank $2$ tensor. A rank $3$ tensor, an element of the tensor product $\mathbb R^3 \otimes \mathbb R^3 \otimes \mathbb R^3$, would then consists of linear combinations of the $27$ pure tensors: $$e_i \otimes e_j \otimes e_k$$ where $i,j,k=1,2,3$. The $27$ coefficients in such a linear combination make up the $3 \times 3 \times 3$ array you mention.
Tensor multiplication is then just given by the good ol' distributive property. For instance, the product of the rank $1$ tensor $2e_1+ 3e_2$ and the rank $2$ tensor $-2(e_1 \otimes e_2) + 2(e_2 \otimes e_3)$ is: $$[2e_1 + 3e_2] \otimes [-2(e_1 \otimes e_2) + 2(e_2 \otimes e_3)]$$ $$-4(e_1 \otimes e_1 \otimes e_2)+4(e_1\otimes e_2 \otimes e_2)-6(e_2 \otimes e_1\otimes e_2)+6(e_2 \otimes e_2\otimes e_3).$$ For two rank $1$ tensors $$ae_1+be_2+ce_3 \text{ and } xe_1+ye_2+ze_3,$$ tensor multiplication gives a rank $2$ tensor whose coefficient matrix (i.e. the matrix whose entries are the coefficients of the $e_i \otimes e_j$ terms) is the product of the matrices $$\begin{pmatrix}a\\b\\c\end{pmatrix} \text{ and } \begin{pmatrix}x&y&z\end{pmatrix},$$ as you alluded to in your question. However, in general there is no simple relation between tensor multiplication and matrix multiplication.
• 1) Thanks for the answer! If I understand correctly, tensors are really the generalization of vectors, they have this same behavior with vectors such that they are the linear combinations of "pure" tensors, like vectors being the linear combination of "pure" basis vectors. I have another question though: The tensor product $e_1 \otimes e_2$ works like the Cartesian product of $e_1$ and $e_2$'s coordinates, right? – Ufuk Can Bicici Jun 22 '14 at 6:30
• 1) I'm not sure what you mean by "works like the Cartesian product of $e_1$ and $e_2$'s coordinates", but if by that you mean you can think of $e_1 \otimes e_2$ as $(e_1,e_2)$, then yes, as long as you keep the additional stipulations I mentioned above in mind. (For instance, $(u,v+w) = (u,v)+(u,w)$.) This is the reason for using a new notation $\otimes$, since an ordinary Cartesian product doesn't satisfy these additional properties. As for 2), it seems you're essentially asking why tensor products are useful/important. This really deserves its own question, or check Wikipedia first. – Santiago Canez Jun 22 '14 at 14:02
• Well, I might lost the actual definition of the tensor product then. In the beginning we were in the 3 dimensional space where we had only regular vectors as we know; defined as the linear combination of three basis vectors $e_1, e_2, e_3$. Then we take the next step to space of rank 2 tensors where all tensors are built as the combination of 9 basis tensors $e_i \otimes e_j$ with $i, j= 1,2,3$. – Ufuk Can Bicici Jun 22 '14 at 17:06
• The $\otimes$ operation we define has to manipulate the basis vectors $e_1, e_2, e_3$ somehow such that it produces $e_i \otimes e_j$s by obeying the stipulations you mentioned. I want to know what "manipulation" does $\otimes$ perform exactly? – Ufuk Can Bicici Jun 22 '14 at 17:07