Is notational compactness in tensors (compared to linear algebra) relevant? In this post you can read:

A matrix is a special case of a second rank tensor with 1 index up and 1 index down. It takes vectors to vectors, (by contracting the upper index of the vector with the lower index of the tensor), covectors to covectors (by contracting the lower index of the covector with the upper index of the tensor)...

Other answers point out the presence of an implied basis in a tensor when expressed as a matrix, as well as the basis transformation rules.
I want to know if it would be correct to illustrate the quote above by saying that, although the operation of a second rank tensor on a vector (or covector) can be expressed in linear algebra form, tensor notation has the advantage of having the order implied in the sequence of covariant and contravariant indices, leading to more compact notation. This advantage is bound to be magnified in higher order operations.
For example, a row vector (covariant or covector) would be operated on by a second rank tensor in matrix form as:
$\begin{bmatrix}a_{1}&\cdots & a_{ m}\end{bmatrix}
\begin{bmatrix}t_{\alpha 1} & \cdots & t_{\gamma n} \\ \vdots &\ddots &\vdots \\ t_{\alpha m} &\cdots & t_{\gamma n}\end{bmatrix}
=\begin{bmatrix}b_{1}\cdots& b_{ n}\end{bmatrix}$
which in tensor notation could be expressed simply as the following index contraction:
$[T^{\alpha}{}_{\gamma}e_\alpha\otimes e^\gamma](\vec a^\top)=T^{\alpha}{}_{\gamma}\vec a_\alpha=T^{}_{\gamma}=\vec b$
Similarly, the same tensor in matrix form can operate on a column vector (contravariant or "vector") as:
$\begin{bmatrix}t_{\alpha 1} & \cdots & t_{\gamma n} \\ \vdots &\ddots &\vdots \\ t_{\alpha m} &\cdots & t_{\gamma n}\end{bmatrix}
\begin{bmatrix}d^1\\\vdots\\d^n\end{bmatrix}
=\begin{bmatrix}f^1\\\vdots \\ f^{m}\end{bmatrix}$
but again this order of operations is encapsulated in tensor notation as index contracture:
$[T^{\alpha}{}_{\gamma}e_\alpha\otimes e^\gamma](\vec d)=T^{\alpha}{}_{\gamma}\vec d^\gamma=T^{\alpha}=\vec f$
$(*)$ Above $\vec f$ was initially $\vec e,$ which makes the OP concordant with the comments. It was changed because of $e$ is the symbol for the basis vectors, leading to confusion.
 A: I prefer to view tensor notation as a sort of “wiring diagram,” and that this makes tensor notation so great. Indeed, I agree with you that the sequence of indices is important, as just because you have a tensor $A^{\alpha\beta}$ it does not mean that this is the same as $A^{\beta\alpha}$ and so the sequence of indices really matters.
Symmetry and antisymmetry
Sequence does not form the only or most critical part of tensor notation for me; my love extends to some other aspects too. But sequence certainly has a certain power: for example we can classify those 2-tensors as the symmetric or antisymmetric, $A^{\alpha\beta} = \pm A^{\beta\alpha}$ respectively, and then we can say that any 2-tensor can be written as the sum of a symmetric and antisymmetric part,
$$A^{\alpha\beta} = \frac12 \big(A^{\alpha\beta} + A^{\beta\alpha}\big) + \frac12 \big(A^{\alpha\beta} - A^{\beta\alpha}\big).$$
When you compare to the matrix notation then this is somewhat lost. For example the Lorentz 4-force in matrix notation (in Gaussian or my favorite units) is
$$\gamma \frac{\mathrm d\phantom t}{\mathrm d t}\begin{bmatrix}E/c\\p_x\\p_y\\p_z\end{bmatrix} = \frac{q}{c}~
\begin{bmatrix}0&E_x&E_y&E_z\\
E_x&0&B_z&-B_y\\
E_y&-B_z&0&B_x\\
E_z&B_y&-B_x&0\end{bmatrix}~
\begin{bmatrix}
\gamma c\\\gamma v_x\\\gamma v_y\\\gamma v_z\end{bmatrix}$$
and you may not believe me, but that matrix in the middle there is actually one of these antisymmetric 2-tensors. It doesn’t look antisymmetric, as a matrix, but it turns out that it has antisymmetry as a tensor. In fact almost every electromagnetism text will write out for you that
$$F_{\mu\nu} = \partial_\mu A_\nu - \partial_\nu A_\mu \Leftrightarrow \begin{bmatrix}0&E_x&E_y&E_z\\
-E_x&0&-B_z&B_y\\
-E_y&B_z&0&-B_x\\
-E_z&-B_y&B_x&0\end{bmatrix},$$and this visibly antisymmetric entity is not actually a proper matrix; it is technically bogus as the thing that comes out of it is a covector, not a vector. Stijn in comments on the original post makes this point a different way; he says that one might view the matrix as simply being a rectangular array of numbers and then this presentation is not technically incorrect. (I would still argue that it facilitates a form of misuse if the resulting column vector is not transposed to a row vector immediately.)
I like matrices a lot, do not mishear me! Matrices represent “naturally” these  [1, 1]-tensors (and [1, 0]- and [0, 1]-tensors) in a particular basis, and there is a trick called the Kronecker product where we can use a larger vector space (treat [2, 0]-tensors as our new “vectors”) and then we can use matrices to represent $[n, n]$-tensors on that larger space of $[n, 0]$-tensors. In addition you can use them to visually depict $2n$-tensor components if you are willing to think of them as just a grid of numbers. In addition, while the only matrices you create in this context are square or straight (i.e. rows or columns), matrices come in a wide variety of rectangles that are useful for things like describing linear regression, or, say, Markov transition steps... lots of other uses than just the geometric vectors that we care about in physics.
Applications to skewed coördinates
In physics, tensor notation gives us a clean way to deal with skewed coördinate systems. So when you are defining unit vectors $\mathbf e_m$ it is really helpful in skewed coördinates where $\mathbf e_m \cdot \mathbf e_n \ne \delta_{mn}$ to invent the dual vectors $\mathbf e^m$ such that $\mathbf e^m \cdot \mathbf e_n = \delta_{mn},$ where $\delta$ here is the normal Kronecker delta $\delta_{mn} = \{1 \text { if } m = n \text{ else } 0\}.$ The dual basis vector to any starting basis vector in some basis is thus the vector that is

*

*(direction) perpendicular to all of the other vectors of that basis, and

*(magnitude) scaled so that the dot product with the starting vector is 1.

Usually in solid state physics courses we give a tiny sliver of this by teaching the dual basis vectors $$\mathbf b_1 = 2\pi~\frac{\mathbf a_2 \times \mathbf a_3}{\mathbf a_1 \cdot (\mathbf a_2 \times \mathbf a_3)}, \text{ etc.,}$$
where if we taught tensor notation much earlier someone would just say $b^i = 2\pi a^i$ or so and possibly nobody would even bother with the explicit definition. Perhaps we would just adopt the $\tau = 2\pi$ convention to save our keystrokes, or, even more ambitiously, perhaps we might take the implicit argument of $1$ as a complex number to be $1 = e^{2\pi i}$ and thus start to write waves as $1^{\mathbf k\cdot\mathbf r - f t}$ or other ways to lighten our notational burden. But the point is that there is no reason for us to be specially teaching this dual basis only to folks doing solid-state physics.
Coördinate-free representation
The above two points hint that tensor notation is secretly geometric and is much deeper than a choice of coördinates. This insight is called “abstract index notation.”
Here’s how this works: we take for granted something called “scalars” and some other thing called “vectors” forming a module over the scalars, and the covectors, which are the linear functions from vectors to scalars, as the first toe-dip into general tensors. We define the set $\mathcal T(m, n)$ to be the set of multilinear functions from $m$ covectors and $n$ vectors to a scalar. So $\mathcal T(0,0)$ is precisely the scalars and $\mathcal T(0, 1)$ is precisely the covectors, whether the co-covectors $\mathcal T(1, 0)$ are precisely the vectors probably requires an extra axiom that in physics we always take for granted, but certainly the vectors are a subset. (A metric is an isomorphism between vectors and covectors and probably suffices to guarantee that the co-covectors are vectors? That gets a bit further into mathematical technicalities than I am comfortable with.)
We can call anything that lives in $\mathcal T(m, n)$ an $[m, n]$-tensor and if we have a metric then we can use that bijection to regard it as a $(m+n)$-tensor, e.g. the metric is naturally a symmetric [0, 2]-tensor or just a 2-tensor, with an inverse. So, like, all of that structure exists before we even start to talk about notations.
The way abstract index notation works is, we make copies of $\mathcal T(m, n)$ for any two disjoint sets of symbols $S_1 \cap S_2 = \emptyset$ such that $|S_1| = m, |S_2| = n$, and each of those copied spaces is a module over our scalars. Each set is a set of symbols: order does not matter as far as “which tensor space are we in?”. And the reason that we create these disjoint copies is so that you can’t add things from one copy to things from another copy; that is a “type error.” To keep track of what terms are allowed we denote a tensor with its usual symbol, plus the $m$ symbols of $S_1$ as superscripts, and the $n$ symbols of $S_2$ as subscripts. So just by looking in $A^{pq}_r$ you can tell that this lives in this space $\mathcal T(\{\textrm"p\textrm",\textrm"q\textrm"\}, \{\textrm"r\textrm"\})$ which is a copy of $\mathcal T(2, 1)$. Since order matters for each tensor but does not matter for the space, one could hypothetically write something like $A^{pq}_r - A^{qp}_r$ and this is not a type error: those are two tensors in the same space and they can be subtracted.
Indices for wiring vectors and covectors together
The indices here are used to therefore do a bunch of things:

*

*To declare a ‘type’ of an operator or a space that it naturally lives in.

*To define the “outer products” by which we can take a tensor in $\mathcal T(a, b)$ and another tensor in $\mathcal T(c, d)$ and form a tensor in $\mathcal T(a+c, b+d)$. Here the indices are used to indicate how the input vectors/covectors are going to be “wired” into the constituent tensors to form the final scalar, and the notation that we use for this is just juxtaposition: $A^{pq}_r B^{\vphantom{p}s}_{\vphantom{r}}$ lives in the space $\mathcal T(\{\mathrm"p\mathrm", \mathrm"q\mathrm", \mathrm"s\mathrm"\}, \{\mathrm" r\mathrm"\})$ and is an outer product of a [2, 1]-tensor and a [1, 0]-tensor, with the index $s$ being wired to that [1,0] tensor and the other indices wired to the [2,1]-tensor. Crucially for the next part, an outer product in its pristine natural state is forbidden to reuse an index symbol in either position, as that would generate an ambiguity with the next point.

*To finally wire together the tensors internally via the operation of index contraction, so that the expression $B^{\alpha\beta\gamma}_{\phantom{\alpha\beta}\gamma}$, because of the repeated $\gamma$ index, now lives in the space $\mathcal T(\{\mathrm"\alpha\mathrm"\},\{\mathrm"\beta\mathrm"\})$ that we copied from the ur-space $\mathcal T(2,0),$ which is the space of multilinear functions that take two covectors and produce a scalar.

The exact geometric meaning of contraction takes a bit more work to specify unambiguously geometrically, but is something like, “we assume an axiom that everything in $\mathcal T(m, n)$ can be written as a sum of terms that are individually in $\big(\mathcal T(1,0)\big)^m \times \big(\mathcal T(0, 1)\big)^n$ and then we take each of these terms and feed the ‘correct’ covectors into the co-covectors to produce scalars, which we can now multiply the remaining tensors by and sum back together.” So there is an implicit appeal to something like coördinate decomposition but it is recast as a totally geometric operation.
In addition to these we have a relabeling isomorphism connecting these different copies of $\mathcal T(m, n)$ that have different label sets, and we can call which we could call $\delta^a_b$ distinct from, but functioning very similarly to, the Kronecker delta above: this can be thought in the $\delta^{\bullet}_\bullet$ case as taking a covector and a vector and applies the one to the other to produce a scalar; or via contraction it can be viewed simply as changing the name of one index to another.
Specific tensors
Maybe the best part for me about tensor notation as wiring is that everything becomes a component to be wired in. So we can introduce particular tensors as part of the notation. The most common is a symmetric inner product and its inverse, $g^{ab}$ and $g_{bc}$ such that $g^{ab}g_{bc} = \delta^a_c.$ This is usually taken to be a canonical isomorphism between the vector and covector spaces, so that $v_a = g_{ab} v^b$ by definition and thus an inner product of a vector with itself looks like $v_a v^a.$ But, 2-spinor calculus instead introduces an “inner product” (it no longer obeys the axioms) which looks like $\epsilon^{AB}$ and is now antisymmetric rather than symmetric.
Penrose graphical notation takes this even further by denoting the upper indices as wires going up and lower indices as wires going down, and the metric tensor is a little half-circle of wire switching an upward-going wire to a downward-going wire and vice versa. We can do this with spinors as long as we are careful to place an arrow on the wire to indicate symmetry vs. antisymmetry.
Or, say we remain in an ordinary $D$-dimensional space with an ordinary inner product tensor, but now we introduce a $[0, D]$-tensor, totally antisymmetric in all pairs of its indices, called the orientation tensor $\epsilon_{abc\dots}$ which you might know in 3 dimensions as the cross product $\epsilon_{abc}$. This is another tensor to be added to the notation. In relativity we get an $\epsilon_{abcd}$ instead, with four wires. It turns out when we apply that to the 2-tensor electromagnetic field above, we get a new 2-tensor which flips the electric and magnetic fields in the above patterns, so that in relativity such antisymmetric 2-tensors are often called “bivectors” because they have these two 3-vector parts, one straightforward on the space/time axes of the matrix and one scrambled up into a cross-product inside the 3x3 space/space part of the matrix, and this orientation allows us to flip which is scrambled and which is straight.
Or, our scalars become scalar fields $\mathcal M \to \mathbb R$ over a manifold $\mathcal M$: and then we can introduce with our resulting vector fields the idea of a spatial derivative or connection operator $\nabla_a$.
Or, we introduce in any of these contexts an explicit basis. Basis vectors are really simple here; one chooses e.g. Greek indices to always be abstract and Roman indices to always be stand-ins for actual concrete numbers, and then introduce some vectors $c^\alpha_1, c^\alpha_2, \dots c^\alpha_D$ as our basis vectors. This prompts inventing the dual vectors above, $$c^{m}_\alpha ~c^\alpha_n = \{1 \text{ if } m = n \text{ else } 0\}$$ by which we might reintroduce a Kronecker delta $\delta^m_n$. Perhaps we combine this with the idea of physical fields above, and we introduce a bunch of scalar “coördinate fields” $C^{1,2,\dots D}$ and then we form our basis covectors as $c_\alpha^n = \nabla_\alpha C^n.$
But the point is that these are all just tensors in the algebra. We can add them as we need them, because the “hard part” of keeping track of different vector/covector arguments to our multilinear functions and wiring them all up together, is now solved.
So, done properly all of your tensor-notation operations become coördinate-free until you explicitly substitute in coördinates later, and symmetry and antisymmetry are strictly coördinate-free statements.
A: The matrix representing  a linear map $A:V\to V$ in a basis ${\bf e}_a$ is given by
$$
A({\bf e}_a)= {\bf e}_b {A^b}_a
$$
exactly as you say. So ${\bf y}= A({\bf x})$, where  ${\bf x}= x^a {\bf e}_a$, becomes
$$
A(x^a {\bf e}_a)= x^a {\bf e}_b {A^b}_a= {\bf e}_b {A^b}_a x^a= y^b {\bf e}_b.
$$
Or, comparing components,
$$
y^b = {A^b}_a x^a
$$
This is just
$$
\left[\matrix{ {A^1}_1 & {A^1}_2 &\ldots \cr {A^2}_1 & {A^2}_2 &\ldots\cr 
\vdots &\vdots&\ddots}\right]\left[\matrix{x^1\cr x^2\cr \vdots}\right]= \left[\matrix{y^1\cr y^2\cr \vdots}\right], \quad \hbox{or} \quad {\bf y}={\bf A}{\bf x}
$$
in matrix notation.
So you are quite correct. Which notation one prefers is a matter of choice.
