Is there hope for Einstein tensor notation in Quantum Mechanics? The Core Question
How would one define and justify the tensor notation for vectors and operators on a separable Hilbert space?
Motivation
Whenever I work with finite-dimensional linear algebra, I find it very illuminating to think about vectors, linear maps, bilinear forms etc. collectively as just various types of tensors. I like how the tensor formalism puts emphasis on representation-free formulations and how even a very complicated multi-linear map can be understood in terms of its individual „indices“.
When I started to learn Quantum Mechanics, I immediately tried to use the tensor notation. At a surface level it seemed to work just fine. One can formally rewrite the bra-ket expressions on left with a „tensor-like“ notation on the right:
$$
\begin{aligned}
  \left< \psi \mid \varphi \right>
  \qquad &\leftrightsquigarrow \qquad
  \overline{\psi^{\,\mu}} \; g_{\overline{\mu} \nu} \; \varphi^\nu
  \\
  \left| \psi \right> = \hat A \hat B \left| \varphi \right>
  \qquad &\leftrightsquigarrow \qquad
  \psi^{\,\mu} = A^\mu_\nu \; B^\nu_\kappa \; \varphi^\kappa
  \\
  W = \left| \psi \right>\left< \psi\right|, \;
  \operatorname{Tr} W = 1
  \qquad &\leftrightsquigarrow \qquad
  W^\mu_\nu = \psi^{\,\mu} \; g_{\overline{\kappa}\nu} \; \overline{\psi^\kappa}, \;\;
  W^\mu_\mu = 1
\end{aligned}
$$
Furthermore, the tensor notation feels more natural when dealing with composite systems, where the state is a tensor product of states of the underlying systems. However, when I tried to put the notation on a more rigorous ground, I learned that the topic is much more difficult and nuanced than I thought.
The Problem
In finite-dimensional spaces, the tensor notation is made possible by several circumstances:
\begin{gather}
V^* \otimes V \simeq \operatorname{Hom}(V, V)
\label{homomorphism} \tag{1} \\[5pt]
T(\mathbf{v}) = T( \; \sum_k v^k \mathbf{e}_k \;) = \sum_k v^k \; T(\mathbf{e}_k)
\label{continuity} \tag{2} \\
\big| \operatorname{Tr}(T) \, \big| < \infty
\label{trace} \tag{3}
\end{gather}
While \eqref{homomorphism} essentially makes sure that all $k$-covariant $l$-contravariant tensors are from the same space, \eqref{continuity} lets us describe all tensors using their coefficients wrt. some basis and \eqref{trace} lets us express all operations on tensors using just tensor product and contraction.
In a sense, neither of these is true for the separable Hilbert space $\mathcal H$ where QM is done. By a basis on a Hilbert space one usually means the Schauder basis, which describes vectors with an infinite series of coefficients. This, combined with the fact that most interesting operators in QM aren't continuous, gives us:
$$
  T(\mathbf{v})
  = T( \; \sum_{k=1}^\infty v^k \mathbf{e}_k \;)
  = T( \; \lim_{N\to\infty}\sum_{k=1}^N v^k \mathbf{e}_k \;)
  \neq \lim_{N\to\infty} T( \; \sum_{k=1}^N v^k \mathbf{e}_k \;)
  = \sum_{k=1}^\infty v^k \; T(\mathbf{e}_k)
$$
The sequence on the RHS might either diverge, or even converge to a different value than LHS (although this is pathological and usually not considered in practise). Condition \eqref{trace} only holds for the so-called trace-class operators (not even the identity is trace-class) and the space $\mathcal H^* \widehat{\otimes} \mathcal H$ (tensor product of Hilbert spaces) is isomorphic to finite-rank operators, a proper subset of $\operatorname{Hom}(\mathcal H, \mathcal H)$.
A Possible Solution? (and more questions)
These unfortunate circumstances mean, that if it's even possible to justify the tensor notation in infinite-dimensional spaces, it can't be by a simple generalization of the finite-dimensional case. So, is it possible to make it work?
The condition \eqref{continuity} seems relatively easy to fix – one just needs to replace the Schauder basis with a Hamel basis. A serious downside to this would be that a typical operator would have an uncountable number of coefficients.
For \eqref{homomorphism} it would be great if one could find a space $\mathcal G$, such that the Hilbert space $\mathcal H$ is embedded in it $\mathcal H \subset \mathcal G$, with the property $\mathcal G \otimes \mathcal G \simeq \operatorname{Hom}(\mathcal H^*, \mathcal H)$. Vectors of the bigger space could then be used to construct operators and bilinear forms. Here, one immediately thinks about the Gelfand triple $\Phi \subset \mathcal H \subset \Phi^*$ – could it be that $\mathcal G = \Phi^*$ for an appropriate choice of rigging? Sadly it appears that no, as the Schwartz kernel theorem says that $\Phi^* \otimes \Phi^* \simeq \operatorname{Hom}(\Phi, \Phi^*)$. Is such a space $\mathcal G$ even possible? If yes, would this approach generalize to tensors of higher degree?
Finally, the problem with \eqref{trace} seems to be there to stay. There is no reasonable way to define the trace of identity, for example. What I'm interested in is whether, given a complicated expression, one can tell a priori which indices are contractible and which will inevitably diverge.
Are these proposed “solutions” any good? Is there any literature that would investigate this topic? Or am I on the wrong path and should I stop wasting my time with tensors in the infinite dimension?
 A: In general, this sounds like a bad idea.
First, the notation thing. When your Hilbert space is $\mathbb R^n$ or $\mathbb C^n$, there is a clearly preferred basis, the canonical basis. The same is kind of true in some infinite-dimensional space, like $\ell^2(\mathbb N)$. But what would you do if your Hilbert space is presented as $L^2[0,1]$? In that case the usual inner product is
$$
\langle \psi\,|\,\varphi\rangle=\int_0^1 \varphi(t)\overline{\psi(t)}\,dt,
$$
which you propose to write as $\overline{\psi^{\,\mu}} \; g_{\overline{\mu} \nu} \; \varphi^\nu$; I cannot imagine a single advantage of doing so.
It is also bad when you want to write operators in terms of coordinates; it has been known for a very long time that thinking of operators on a Hilbert space as infinite matrices is not very fruitful.
Writing the trace in coordinates is not nice either, as it is not defined for all operators: for a non-trace-class operator, the infinite sum of its diagonal entries will usually differ with respect to different bases. The expression $W_\mu^\mu$ is defined every time you have a basis, but it will give you different values for different bases when $W$ is not trace-class. So $\operatorname{Tr}(W)$ is a much better notation, as it emphasizes that it can only be used where it can be reasonably defined.
I think that it is much easier to glance that $W=|\psi\rangle\,\langle\psi|$ is a rank-one operator from this notation than from $ W^\mu_\nu = \psi^{\,\mu} \; g_{\overline{\kappa}\nu} \; \overline{\psi^\kappa}$. And this is not me defending the bra-ket notation, which I think is bad and would be better replaced with $\psi^*\varphi$ instead of $\langle \psi\,|\,\varphi\rangle$ and $\psi\varphi^*$ instead of $|\psi\rangle\,\langle\varphi|$.
As for tensors, the Hilbert space tensor product $H^*\otimes H$ is canonically identified with the Hilbert-Schmidt operators. This is far less than the whole algebra $B(H)$ of bounded operators on $H$. If you want the norm of $H^*\otimes H$ to have any relation with that of $H$, you'll get that $H^*\otimes H$ is separable, while $B(H)$ isn't in the norm topology. This doesn't mean that it is impossible to somehow achieve what you want, but it is hard for me to see what could be achieved.
Your "fix" to your condition (2) is worse than a "serious downside". Basically what you want is to write things in such a way that continuity is invisible. In other words, you want to drop all topological considerations; for example, now you cannot say that if $\varphi_n\to\varphi$, then you have in coordinates $(\varphi_n)_\mu\to\varphi_\mu$. The "Schauder" bases that you mention, in a Hilbert space, are way more than that. They are "orthonormal bases". And, ironically since you want to avoid them, they are good precisely because they are the closest you can get to the finite-dimensional case. Orthonormal bases allow you to easily express the norm of a vector in terms of its coefficients. They allow you to easily define isomorphisms of Hilbert spaces: the unitary operators, which incidentally play a crucial role in QM. Dealing with orthonormal bases, unitaries are simply the operators such that their columns are orthonormal bases.
Just a few thoughts off the top of my head.
