# Is there a fundamental problem with extending matrix concepts to tensors?

We are familiar with the theory of matrices, more specifically their eigen-theorems and associated decompositions. Indeed singular value decomposition generalizes the spectral theorem for arbitrary matrices, not just square ones. Now it only seems natural to extend this idea of 2 dimensional array of numbers to higher dimensions, i.e. tensors. But as soon as we do this, everything breaks down.

For example even the notion of rank of matrix (which we all agree to be minimum of either column rank or row rank for a matrix) seems to be conflated when it comes to tensors. The Wikipedia page seems to use degree, order and rank of a tensor synonymously (understandably due to different terminology used in different fields, but somewhat annoying nevertheless).

"The order (also degree or rank) of a tensor is the dimensionality of the array needed to represent it, or equivalently, the number of indices needed to label a component of that array."

Also for example, the very familiar concept of eigenvalues or vectors also flies out the window (though people have defined them for super-symmetric tensors). So my question is this:

What is the fundamental reason why the "nice" theorems we have in the matrix case do not extend to the case of tensors?

I can think of a couple:

• Tensors exhibit the phenomenon of rank jumping as well as field dependence; which would imply the usual rules of analysis need to be re-examined when dealing with them.
• A large class of matrices are groups, so tools from abstract algebra are available to deal with them.
• Matrices can be viewed as operators from one space to another unambiguously, whereas viewing a tensor as an operator between spaces can get confusing very quickly.

I know there are extensions to SVD for tensors, for example Tucker decomposition, HOSVD, etc; so I am not claiming it can't be done. I also understand (somewhat) that mathematicians prefer to study tensors abstractly or using differential geometry and forms. I am just curious as to why the results generalizing the concepts form the matrix case are many; what is the underlying cause for a lack of unifying framework. The above reasons seem valid roadblocks, but do they hint at something more fundamental?

• "rank" is probably the most overloaded term in multilinear algebra, I can think of at least three different meanings of the "rank" of a tensor. I would be careful as to which one you actually mean. – Joppy May 2 '18 at 13:10
• I would think the most natural extension of rank of a matrix to rank of a tensor is in the SVD sense, but then it goes back to the question, what’s SVD really for tensors? – ITA May 2 '18 at 13:16
• Your question already has the rank of an operator as the dimension of the image of the operator, and the rank of a tensor as the number of indices in the tensor. These are totally different concepts - which one do you want to consider? – Joppy May 2 '18 at 13:18
• I am not considering rank as the number of indices; that was a quote from Wikipedia to show that the concept is overloaded/conflated as you rightly mentioned. – ITA May 2 '18 at 13:22
• I think you can extend some theorems of matrices to tensors. For instance the rank. For matrices, the rank of a matrix can be defined as the number of linear independent rows or columns ( rank 1 vectors), while for three-order tensors it can be defined as the number of linear independent rank 1 slices ( rank 1 matrices). – Alex Silva May 2 '18 at 13:59

I'd argue that the main point is that a matrix is (among other things) a linear map while a tensor is not.

All matrix decompositions can be seen as choosing particular bases in the domain and codomain of the linear map:

• For example LU decomposition can be written as $$A = LU = \tilde L \begin{pmatrix}I_r & 0\\0 & 0\end{pmatrix}\tilde U$$ where $r$ is the rank and $\tilde L$ and $\tilde U$ are slight modifaction of the original LU decomposition such that they become invertible, i.e. $\tilde U$ comes from a basis in the domain and $\tilde L$ from a basis in the codomain.
• SVD is inherently a choice of bases $A = U\Sigma V^T$ with orthonormal $U$ und $V$…
• Same for spectral decomposition of symmetric $A$ (a special case of the SVD).

Ok, a matrix is also a bilinear form and a tensor is a multilinear form and, actually, this similarity can be used to define eigenvectors of tensors, but, I'd agrue that the point of view that a matrix represents a linear map is more fundamental. There a several matrices $A$ that represent the same bilinear form (e.g. $A$, $A^T$ and $(A+A^T)/2$ all represent the same bilinear form), while the linear map is unique (up to the choice of bases).

I have the impression that these notions for matrices that rely on the view as bilinear forms generalize much easier to tensors (multilinear maps) as the notions that rely on the view as a linear map.

• I agree with you .. but so far I have only come across eigenvalues of tensors defined for super-symmetric tensors, that is tensors whose elements remain the same under every permutation of all of its indices, and in fact this "generalization" relies on the the bijection between the space of homogeneous polynomials and super-symmetric tensors to define eigenvalues. If it was just a matter of viewing tensors as multi-linear maps the generalization of eigenvalues wouldn't require such a heavy restriction of super-symmetry. – ITA May 7 '18 at 13:28

Having in mind that for a vector space $V$ there is another vectorspace $$V^*=\{V\to\Bbb F:\mbox{which are linear}\}$$ dubbed the dual then tensors are:

1) covariant tensors which are multilinear maps $$V\times...\times V\to\Bbb F,$$ 2) contravariant tensor, multilinear maps $$V^*\times...\times V^*\to\Bbb F$$ and 3) mixed tensors as multilinear maps $$V\times...\times V\times V^*\times...\times V^*\to\Bbb F.$$

In each stage one is obliged to use not only bi-indexed quantities as $$A_{ij},\ B_i{}^j,\ C^i{}_j,\ D^{ij}$$ identified as the components of a tensor and which correspond to entries of matrices. Each of those for the four possible types $$V\times V\to\Bbb F,$$ $$V\times V^*\to\Bbb F,$$ $$V^*\times V\to\Bbb F,$$ $$V^*\times V^*\to\Bbb F,$$ of bilinear maps.

But there are also tri-indexed components as $$T^{ijk}, U^{ij}{}_k,...etc$$ where you begin to have a generalization of matrices.

• a paraphrasis would be: ... tensors are like any other machine. They're either a benefit or a hazard. If they're a benefit, it's not a problem. – janmarqz May 8 '18 at 17:31