25
$\begingroup$

How would you explain a tensor to a computer scientist? My friend, who studies computer science, recently asked me what a tensor was. I study physics, and I tried my best to explain what a tensor is, and I said something along the lines of "a mathematical object that is described in between the mappings of vector spaces", and he wasn't quite about that definition. I understood why, since it is a pretty wordy definition I gave, so I decided to give a more, down to earth, definition, describing a tensor as some array, with n-dimensions. However, he was still kind of confused by this. Can anyone synthesise a decent definition, tailored to a computer scientist's understanding?

$\endgroup$
16
  • 11
    $\begingroup$ How would you explain what tensors are , to Newtonian Physicists or Quantum Physicists or Einsteinian Physicists ? $\endgroup$
    – Prem
    Feb 11 at 17:48
  • 17
    $\begingroup$ its just a multilinear map, this is all $\endgroup$
    – Masacroso
    Feb 11 at 18:01
  • 21
    $\begingroup$ A computer scientist should have no trouble to understand what multi-dimensional arrays are: v[i] vector, m[i,j] matrix, t[i,j,k] 3rd order tensor, and so on. A cemetery for numbers. Physics deals more often with tensor fields on manifolds but calls them sloppily tensors as well. $\endgroup$
    – Kurt G.
    Feb 11 at 18:52
  • 9
    $\begingroup$ Incidentally, the question is somewhat fraught with the use of "tensor" in some CS circles to refer to just a multi-dimensional block of numbers (the generalization of an array) that may have no implicit structure, or may have a structure completely different from what mathematicians or physicists might expect. $\endgroup$
    – Brian Tung
    Feb 11 at 19:14
  • 19
    $\begingroup$ In machine learning, a “tensor” is simply a multidimensional array of numbers. Important to note that what mathematicians call a tensor is different than what machine learning people call a tensor. $\endgroup$
    – littleO
    Feb 11 at 20:35

7 Answers 7

31
$\begingroup$

A sound mathematical definition of a tensor product is not trivial, certainly not trivial to understand. Then working with tensors in addition also requires to grasp the concept of a vector space and its dual, and so on. That is not an obvious, or productive, path to take for a computer scientist, even though only an understanding of all this will reveal the why behind tensor operations that can be found in software libraries.

From a CS perspective I would explain that tensors are, indeed, multi-dimensional arrays plus a number of basic operations on them that have turned out to be very useful in a lot of applications. For example, most of the operations in a popular library as numpy can be constructed from the following building blocks (not an exhaustive list):

  • Reordering of indices.
  • Applying a binary operator $\star$ to arrays with equal dimensions: $(A \star B)[I] = A[I] \star B[I]$.
  • Free tensor product of arrays: $(A\otimes B)[I,J] = A[I] \cdot B[J]$.
  • Introducing additional indices of any dimension: $A’[I,J] = A[I]$.
  • Contraction of two indices with equal dimension: $A’[I] = \sum_k A[k, k, I]$.

Of course, there are also some not so trivial operations on certain tensors, such as inversion or $QR$ factorization, and so on. However, understanding how a library supports the operations above will make it much easier to understand. The often countless functions that a library offers are mostly just clever and optimized combinations of these basic operations. For example, a matrix multiplication is a combination of a free tensor product and a contraction. The trace of a square matrix is a contraction. And so on.

$\endgroup$
3
  • 5
    $\begingroup$ +1. “Multi-dimensional arrays” is clearly the right first approximation, as all answers agree — but this is the only answer giving good suggestion for what to add to that. For a mathematician or physicist, the key next points are all linear-algebraic — how they transform under change of co-ordinates, or the basis-independent characterisation of the tensor product of vector spaces — and the “natural operations” here are a good way of getting some of that across, without assuming much linear-algebra background. $\endgroup$ Feb 12 at 10:36
  • 10
    $\begingroup$ From my perspective as a mostly CS person, I think an (almost the most) important point to understand is that in code you never have "just a tensor" you always also have an implicit basis in which you represent your tensor. This matters a lot because (1) the basis is almost never found in code. It's often, if ever, found in documentation (2) most n-ary tensor operations in a software library assume all the bases you pass in are the same. It is this assumption that makes "just think of them as arrays" work. $\endgroup$ Feb 12 at 10:51
  • $\begingroup$ @WorldSEnder Thanks for your comment. That is a valuable addition. $\endgroup$
    – WimC
    Feb 28 at 6:20
15
$\begingroup$

If your friend is a computer scientist, then he should be well acquainted with the concept of scalars, vectors, and matrices. I see a tensor as a generalization of scalars, vectors, and matrices to higher dimensions. In other words, a tensor is a multi-dimensional array.

A scalar is a single number. You can view a scalar as a special case of a vector, that is, a vector with a single dimension of size $1$. So, a vector is a generalization of a scalar.

A vector is an ordered collection of numbers in a single dimension. You can view a vector as a special case of a matrix, that is, a matrix with a single dimension of any size $n \ge 1$. So, a matrix is a generalization of a vector.

But a matrix is an array restricted to two dimensions, each with its own size. A tensor extends the notion of a matrix analogous to how a vector extends the notion of a scalar and a matrix extends the notion of a vector. A tensor can have any number of dimensions, each with its own size. A $3$-dimensional tensor can be visualized as a stack of matrices, or a cuboid of numbers having any width, length, and height. A $4$-dimensional tensor can be visualized as a sequence of cuboids, each having a fixed width, length, and height. A $5$-dimensional tensor can be visiualized as a matrix in which every entry is a cuboid, and so on. I think the below image is a nice summary...

enter image description here

Notice a $1$-dimensional tensor is the familiar vector, and a $2$-dimensional tensor is the familiar matrix.

Finally, in the same way scalars, vectors, and matrices can be multiplied by one another to map objects between vector spaces (or to map objects in the same vector space), multiplication with a tensor is no different. There are additional nuances and ideas related to the use and functionality of tensors (see response of WimC for a concise summary), but I think the explanation I provided will get your friend off to a good start.

$\endgroup$
3
  • 6
    $\begingroup$ While this is fine as far as it goes, I suspect OP is trying to convey more about tensors than their basic shape. If the CS student asked "What distinguishes a tensor from just a multi-dimensional block of numbers?" what would you answer, I wonder? $\endgroup$
    – Brian Tung
    Feb 11 at 21:12
  • 4
    $\begingroup$ I didn't get the impression OP wanted a comprehensive and detailed explanation of tensors. It seems he was trying to give a more intuitive description. For instance, he described a tensor as an $n$-d array just as I did, but with no success. If his friend cannot grasp that, then I doubt he will grasp anything more complex. That is why I thought better to reinforce and focus on its description as an $n$-d array instead of going into the weeds. To answer your question, I would need more room than is available in the comments. I suggest you post your question to the SE. $\endgroup$ Feb 11 at 21:48
  • 3
    $\begingroup$ @BrianTung At that point, I would politely point the CS student to the nearest comprehensive PDF manual of the APL programming language. APL is what made tensors really "click" for me. Most of the interesting tensor operations are just built-ins in the array programming world. $\endgroup$ Feb 13 at 4:14
9
$\begingroup$

When telling a computer scientist about a mathematical object, it may be helpful to first specify the type of this object. If the object is a function, what are the types of its inputs and outputs? If the object is a tuple, what are the types of its coordinates?

What might make answering this question difficult is that there are several different accepted definitions of tensors, and these definitions yield different answers to the question "What is the type of a tensor?" However, one can prove that there is a natural correspondence between these definitions.

As an analogy, think about how, given an $n$-dimensional vector space $V$ over a field $F$, there is a natural correspondence between linear transformations from $V$ to $V$ and $n \times n$ matrices of elements of $F$. Because of this correspondence, we often think about linear transformations and matrices interchangeably.

Note that this "natural correspondence" depends on a choice of basis for $V$; that is, we do not have a natural way to identify linear transformations with matrices unless we have specified a basis for $V$.

Two definitions of tensors are the following:

  • Given a vector space $V$ over a field $F$, a $(p,q)$-tensor is a multilinear map $T: \overbrace{V^* \times \dots \times V^*} \times \overbrace{V \times \dots \times V }\to F$, where $V^*$ denotes the dual space of $V$, and a multilinear map is a map that is linear in each coordinate.

  • Given an $n$-dimensional vector space $V$ over a field $F$ with basis $B$, a $(p,q)$-tensor is a $\overbrace{(p+q) \times \dots \times (p+q)}^n$ array of elements of $F$.

Just as one can prove that there is a natural correspondence between linear transformations and matrices, one can prove that there is a natural correspondence between multilinear maps and multidimensional arrays, which means that these definitions are essentially interchangeable.

There are applications of tensors to computer science. I do not know much about these applications, but I believe they arise when data is stored in a multidimensional array and one wants to study the multilinear-algebraic properties of this array.

$\endgroup$
1
  • $\begingroup$ Re type theory: There is a tendency to conflate (semantic) types with representations (syntactic types). The (semantic) type of an object describes its role in the broader system - what operations it can perform, what relations it can participate in, what invariants it upholds, etc. A syntactic type or representation, by contrast, describes how the object is constituted and formally defined. Unfortunately, when mathematicians try to explain mathematical concepts to me, I find that they always focus way too much on the representation, and not enough on the semantics, so I get lost in the noise. $\endgroup$
    – Kevin
    Feb 14 at 5:55
5
$\begingroup$

For a computer scientist:

A tensor is to a vector (or matrix) what a factory[1] is to an object.

An n-dimensional tensor accepts the coordinate system of your screen, and returns an n-dimensional array that represents a geometric quantity that is independent of which coordinate system you chose. This geometric quantity is always the "same" quantity, but its representation changes because your coordinate system changed.

So all tensors look like n dimensional arrays (vectors, matrices, etc), but to qualify as a tensor the numbers in that array need to change consistently when you chose a different coordinate system.

Example:

Consider the 1-dimensional tensor that returns a vector of length one pointing straight up from the bottom of your screen. If you choose the coordinate system where the y axis increases from the bottom of the screen to the top, then this tensor will return the representation $$[0, 1]$$. However, if you choose the coordinate system where the y axis decreases from the bottom of the screen to the top, then this tensor will return $$[0, -1]$$ instead. Different coordinate systems mean different numbers in your array, but they represent the same geometric thing! So it's a valid tensor.

As a counter example, consider a "factory" that always returns $$[0, 1]$$ no matter which coordinate system it is given. In the first coordinate system mentioned above this represents a vector that points upwards. But in the second coordinate system it represents a vector that points downwards. It's not a tensor because the array of numbers it gives for different coordinate systems have a different geometric meaning (the values in the array failed to transform correctly).

For 2d examples, you can consider:

  • a clockwise rotation of 90 degrees. In different coordinate systems it'll have different matrix representations,but they'll always do the same thing to vectors (namely rotate them 90 degrees clockwise). This is a covariant tensor because if you scale your coordinate system units by a factor of k the entries in your matrix will also scale by the same factor of k
  • a measurement of the length of a vector. It looks like a symmetric 2d matrix. It's a contravariant tensor because if you scale your coordinate system units by a factor of k the entries in your matrix will scale by a factor of $\frac{1}{k}$
  • a transformation which projects vectors onto the x axis and then rotates them towards the positive y axis. This will look like a 2d matrix but it's not a tensor, since applying it to the same vector in different coordinate systems will yield different results.
$\endgroup$
3
$\begingroup$

(Answer from a software engineer who has followed a video series on tensors and who would share his personal experience).

I think it is a mistake to assume that computer scientists only think in terms of concrete data structures like arrays. Although a tensor can be represented by a multi-dimensional array, we know that isn't what one is.

Computer languages have plenty of methods for representing abstract behaviours (e.g., interfaces and abstract classes) and many pieces of software contain complex abstractions and behaviours that are comparable to what you might see in mathematics. The difference is that software tends to be far more verbose and descriptive. Calling a function ƒ in mathematics is normal. Calling a function f in your program (even ASCII f) will get you a comment in the code review, plus a request to specify the parameters, return value and types and a lengthy structured comment that can generate API documentation.

(And before you downvote - I don't mean to suggest that mathematics is not rigorous; simply that the form in which it is presented on a whiteboard in a video lecture is far more concise than what a programmer would be used to seeing in their IDE).

So, I have often wondered whether/which mathematical concepts can be redrafted using programming languages. Maybe it won't work for everything (summing to infinity is not recommended) but there must be many areas where it is possible.

I haven't thought about this further (sorry - this isn't a full answer) but in general, the computer scientist brain can handle abstractions, but is used to seeing things in a far more verbose form than the mathematician with far fewer implied or inferred constraints.

$\endgroup$
3
  • 1
    $\begingroup$ "So, I have often wondered whether/which mathematical concepts can be redrafted using programming languages." Tensors certainly can be expressed in a programming language with sufficiently powerful type system. I gave it a shot with hackage.haskell.org/package/linearmap-category $\endgroup$ Feb 13 at 14:50
  • 1
    $\begingroup$ Regarding the same sentence leftaroundabout is commenting on, there's one sense in which the answer is "basically all math". See the Curry-Howard correspondence and the existence of proof assistants. Also, working with an infinite sum in-and-of-itself is not impossible; if the terms can be generated by some algorithm then a computer program ought to be able to handle the sum in full generality as well. $\endgroup$ Feb 13 at 15:01
  • 1
    $\begingroup$ I think bringing a different perspective deserves an upvote, not a downvote. About the sentence starting with "Calling a function", there are two things: a problem of notations as many objects are used in a CS program, making descriptive names a necessity. Also, rigourously, a function in mathematics is an ordered triple, domain, codomain, and the relation (the "formula"), so parameters, return values are covered. It is mostly a high school/precalculus thing to leave domains and codomains implicit. $\endgroup$
    – Taladris
    Feb 13 at 23:47
1
$\begingroup$

Tensors are composed of vectors and they have Ranks

A Tensor of rank $\left[0\right]$: is a Scalar because it has no direction, and no components.

A Tensor of rank $\left[1\right]$: has $1$ index and $1$ basis vector, per component. An example of a rank $\left[1\right]$ Tensor in $3D$ would be: $$A= \left[A_x, A_y, A_z\right]$$

The indexes $x,y,z$ could represent directions of forces.

A Tensor of rank $\left[2\right]$: in $3D$ has $2$ indexes and $9$ sets of $2$ basis vectors giving $9$ total components:

$$A = \left[A_{xx}, A_{xy}, A_{xz}, A_{yx}, A_{yy}, A_{yz}, A_{zx}, A_{zy}, A_{zz}\right]$$

You might use a rank $2$ Tensor to represent the forces inside a solid object. The $1{st}$ index would represent the surface area pointing in the $x$, $y$, or $z$ direction. The $2_{nd}$ index would represent the force on that surface in the $x$, $y$, or $z$ direction.

A Tensor of rank $\left[3\right]$: in $3D$ has $3$ indexes and $27$ sets of $3$ basis vectors giving $27$ total components:

$$A = \left[A_{xxx}, A_{xyx}, A_{xzx}, A_{yxx}, A_{yyx}, A_{yzx}, A_{zxx}, A_{zyx}, A_{zzx},...,A_{zzz}\right] $$

Notice that Tensors can fit nicely in a multi-dimensional array, where indexing by pointers is natural.

$\endgroup$
1
$\begingroup$

The answer is in using multi-array, but in a specific way. By using the array dimensions to represent contravariant and covariant indices, you can effectively distinguish between different types of tensors and their corresponding transformations.

Start with

$\operatorname{int}[1,3]$: This represents a 1-rank tensor with contravariant indices (row indices)

$\operatorname{int}[3,1]$: This represents a 1-rank tensor with covariant indices (column indices)

Then up

$\operatorname{int}[1,3][1,3]$: This represents a 2-rank tensor with contravariant indices (row indices) for both dimensions.

$\operatorname{int}[1,3][3,1]$: This represents a 2-rank tensor with contravariant indices (row indices) for the first dimension and covariant indices (column indices) for the second dimension.

$\operatorname{int}[3,1][1,3]$: This represents a 2-rank tensor with covariant indices (column indices) for the first dimension and contravariant indices (row indices) for the second dimension.

$\operatorname{int}[3,1][3,1]$: This represents a 2-rank tensor with covariant indices (column indices) for both dimensions.

And so on for higher ranks/dimensions. (All examples use size 3, but you can have any size, and of course any type not just integers.)

This notation aligns with the mathematical representation of tensors and allows for easy interpretation of the tensor's structure and transformation properties.

It's a concise and intuitive way to represent tensors in, for example, C#, and it provides clarity regarding the contravariant and covariant nature of tensor indices.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .