How to rigorously understand continuous bases? In Quantum Mechanics it is quite common to see the idea of a continuous basis of a Hilbert space. In truth if $\mathcal{H}$ is the state space of a quantum system and if $X : U\subset \mathcal{H}\to \mathcal{H}$ is the position operator then it is quite usual to consider the set of eigenvectors $\{|x\rangle : x \in \mathbb{R}\}$ indexed by the eigenvalues $x\in \mathbb{R}$.
In that case, Quantum Mechanics books usually assume that one can write for any $|\psi\rangle \in \mathcal{H}$
$$|\psi\rangle = \int_{-\infty}^{\infty}|x\rangle\langle x|\psi\rangle  dx.$$
And also assume we have the completeness relation
$$\int_{-\infty}^{\infty}|x\rangle \langle x| \ dx = I,$$
being $I$ the identity operator and considering $|x\rangle\langle x|$ to be the operator defined by
$$(|x\rangle \langle x|)|\psi\rangle = |x\rangle \langle x|\psi\rangle.$$
Although this works and gives the correct results, I can't understand what is going on here. One of the points is that the integral involved here is quite strange. We are integrating $f(x) = |x\rangle \langle x|\psi\rangle$ but $f : \mathbb{R}\to \mathcal{H}$ and I'm not sure how this integral is defined. 
Also, I know that for bounded operators with discrete spectrum the set of eigenvectors does really form a basis in the usual sense, but here things are quite different.
How does one rigorously understand those basis? What is the correct definitions behind all of this?
 A: The Dirac formalism is not correct for general selfadjoint operators. It's not true even for the specific operators of Quantum.
For example, consider a periodic problem on $[-\pi,\pi]$. The opertor $L=-\frac{d^{2}}{dx^{2}}$ has a two-dimensional eigenspace spanned by $\{ e^{inx},e^{-inx}\}$ for each eigenvalue $\lambda = 1^{2}, 2^{2}, 3^{2},\cdots$ and has a one-dimensional eigenspace spanned by $\{ 1 \}$ for eigenvalue $\lambda=0$. Ignoring the fact that the expansions cannot be written as a integral without some type of delta function, there is still the issue of dimension of the eigenspaces.
The next case that you've probably encountered would be the non-relativistic Hydrogen isotope (no neutron.) The bound states corresponding to different orbits are of varying dimension. You might argue that using separation of variables one can reduce to one-dimensional operators that might have something close to a Dirac representation, but separation of variables is not possible in more general cases. How would you represent the Hamiltonian in this case using Dirac formalism? There are so many adaptions required that it's hard to think of this simple case as being correct in any modified Dirac form without appealing to separation of variables.
Dirac modeled his representation after the ODE representations for one-dimensional Sturm-Liouville operators
$$
            Lf = -\frac{d}{dx}\left(p(x)\frac{df}{dx}\right)+qf.
$$
For example, on a semi-infinite interval such as $[0,\infty)$ where $p$ is contnuously differentiable and non-vanishing, and where $q$ is continuous, but possibly unbounded as $x\rightarrow\infty$, you can impose an endpoint condition at $0$ of the form $Af(0)+Bf'(0)=0$ where $A$, $B$ are real and not both $0$, and the eigenfunctions $\varphi_{\lambda}(x)$ satisfying fixed endpoint condition $\varphi_{\lambda}(0)=-B$, $\varphi_{\lambda}'(0)=A$, and you'll get an expansion
$$
    f(x) = \int_{-\infty}^{\infty}\left(\int_{0}^{\infty}f(y)\varphi_{\lambda}(y)dy\right)\varphi_{\lambda}(x)d\rho(\lambda).
$$
Allowing for a general measure $\rho$ (or, equivalently, using a Riemann-Stieltjes integral with respect to a function $\rho$ of bounded variation on finite intervals), you do get what Dirac had in mind. This is what Dirac was working from. Further, you get Parseval's equality:
$$
          \int_{0}^{\infty}|f(x)|^{2}dx = \int_{-\infty}^{\infty}\left|\int_{0}^{\infty}f(y)\varphi_{\lambda}(y)dy\right|^{2}d\rho(\lambda)
$$
I think you can convince yourself that Dirac's notation is lifted from such a case, and that it can be interpreted as being correct in such a case. This formalism can be extended with a little care to all of $\mathbb{R}$ by patching representations on $(-\infty,0]$ with $[0,\infty)$. And there is no doubt why Dirac's notation gives you something correct for the cases where separation of variables applies and where the operators become classical differential operators.
For general selfadjoint operators, Dirac notation is not particularly good. A general selfadjoint operator $A$ can be represented as a spectral integral proposed by von Neumann in the form
$$
            Af = \int_{-\infty}^{\infty}\lambda dE(\lambda)f.
$$
Notice that von Neumann's notation (which is what I have given above) is related to Quantum as well, for which he was offering a rigorous alternative in the notation of Quantum. What is happening in this integral is that $E(\lambda)$ is an orthogonal projection for each $\lambda \in\mathbb{R}$, and the space onto which $E(\lambda)$ projects is increasing with $\lambda$. The underlying space is some complex Hilbert space $\mathcal{H}$, and
$$
    E(\lambda_1)\mathcal{H} \subseteq E(\lambda_2)\mathcal{H},\;\;\;\lambda_1 \le \lambda_2.
$$
The fact that $E(\lambda)$ is an orthogonal projection is equivalent to
$$
              E(\lambda)^{2}=E(\lambda)^{\star} = E(\lambda).
$$
Because $E$ is increasing, $E(\lambda_2)-E(\lambda_1)$ is an orthogonal projection, and
$$
     A(E(\lambda+h)-E(\lambda))f \approx \lambda(E(\lambda+h)-E(\lambda))f
$$
In infinitesimal notation:
$$
                     AdE(\lambda)f = \lambda dE(\lambda)f,
$$
which is correct in an integral sense
$$
                       A\int_{-\infty}^{\infty}dE(\lambda)f=\int_{-\infty}^{\infty}\lambda dE(\lambda)f,\;\;\; f \in \mathcal{D}(A).
$$
And you have rigorously correct Parseval identities:
$$
                  \left\|\int_{-\infty}^{\infty}dE(\lambda)f\right\|^{2} =\int_{-\infty}^{\infty}d\|E(\lambda)f\|^{2},\\
          \|Af\|^{2} = \left\|\int_{-\infty}^{\infty}\lambda dE(\lambda)f\right\|^{2} = \int_{-\infty}^{\infty}\lambda^{2}d\|E(\lambda)f\|^{2}.
$$
This version of the Spectral Theorem for selfadjoint linear operators holds very generally on any Hilbert space, and it dates back to the 1930's. I would encourage anyone serious about Quantum Mechanics to become familiar with it. Attractive as it may be, Dirac's formalism is not adequate for a foundational approach.
