# When to *say* “tensor” instead of “matrix”?

The difference between a tensor and a matrix is a subtle but important and well discussed point.

In the context of a vector space $$V$$ over a field $$F$$ (and its dual $$V^*$$), briefly put, a $$(p, q)$$-tensor $$T$$ is a multilinear map $$\ {V^*}^p \times {V}^q \to F\$$.

Any basis we choose for $$V$$ and $$V^*$$ can be inherited by $$T$$, and a matrix $$M$$ is a representation of $$T$$ in that particular basis. Had we chosen a different basis, the matrix representation of $$T$$ would be some different $$M' \neq M$$, while $$T$$ is just $$T$$. As a mundane example on $$\mathbb{R}^2$$ for a $$(1, 0)$$-tensor, the idea is just that $$\ \begin{bmatrix}2 \\ 3\end{bmatrix} \neq \begin{bmatrix}1 \\ -2\end{bmatrix}\$$ while $$\ 2e_1 + 3e_2 = e'_1 - 2e'_2\$$. One consistent basis must be used to write a matrix equation, while a tensor equation (like the latter, usually shown in Einstein notation) shows equality of tensors, which holds true regardless of basis.

In practice however, I find literature commonly saying "matrix" where "tensor" would be more appropriate, making me doubt my understanding. Take for example the "covariance matrix" in probability theory. Covariance is conceptualized as the ellipse in the following geometric depiction: This geometric object is invariant to our choice of basis (i.e., the ellipse exists regardless of how you draw your grid-lines), and thus is best modeled with a tensor. Everything I've ever seen done with a covariance "matrix" has been tensorial, from the $$R^{-1} T R$$ style transformations used in principal component analysis (typical of $$(1, 1)$$-tensors) to the contraction to $$\mathbb{R}$$ used in the exponent of the multivariate Gaussian distribution (where the covariance "matrix" is literally used as a bilinear map $$V^* \times V \to \mathbb{R}$$).

Meanwhile, an extremely similar object, the "inertia tensor" from rigid-body mechanics, which describes the spread of mass (instead of probability), is so purposefully called a tensor that many mechanics books include a section explaining the very difference I briefed above. Surely there is a naming inconsistency here?

Then, probably due to the popular software "TensorFlow", I see the matrices used in neural networks being called "tensors" by the machine learning community, even when there is no discernible basis-independent abstract object being described by these matrices. They're just linear mappings, and in most cases aren't even endomorphisms.

I think many people trying to understand tensors can benefit from clarifying these semantics. When should we really say "tensor"? Is it pretentious to say "covariance tensor"? Do any theories utilize a square matrix that isn't just an order-2 tensor in a particular basis? (Perhaps the Jacobian matrix?) Is there any kind of indicative mathematical expression or action that makes you go "yup, that matrix is just representing a tensor - the important object here is the tensor itself."?

There are several meanings of the term "tensor". For example, this context is close to your multilinear-map idea. But to a physicist, or differential geometer, a tensor satisfies a specific transformation law. Suppose we write it in two coordinate systems, one with lower case indices, the other with upper case; then a rank-$$(p,\,q)$$ tensor $$T$$ satisfies$$T^{a_1\cdots a_p}_{\quad\qquad b_1\cdots b_q}=\frac{\partial x^{a_1}}{\partial x^{A_1}}\cdots\frac{\partial x^{a_p}}{\partial x^{A_p}}\frac{\partial x^{B_1}}{\partial x^{b_1}}\cdots\frac{\partial x^{B_q}}{\partial x^{b_q}}T^{A_1\cdots A_p}_{\quad\qquad B_1\cdots B_q},$$with summation over repeated indices. For example, in a $$2$$-dimensional coordinate system a rank-$$(1,\,0)$$ tensor obeys$$T^a=\frac{\partial x^a}{\partial x^A}T^A=\frac{\partial x^a}{\partial x^\mathrm{I}}T^\mathrm{I}+\frac{\partial x^a}{\partial x^\mathrm{II}}T^\mathrm{II},$$if the Roman values $$\mathrm{I},\,\mathrm{II}$$ are the two options for $$A$$.