In Quantum Mechanics one often deals with wavefunctions of particles. In that case, it is natural to consider as the space of states the space $L^2(\mathbb{R}^3)$. On the other hand, on the book I'm reading, there's a construction which it's quite elegant and general, however it is not rigorous. For those interested in seeing the book, it's "Quantum Mechanics" by Cohen-Tannoudji.
The book proceeds as follows: the first postulate of Quantum Mechanics states that for every quantum system there is one Hilbert space $\mathcal{H}$ whose elements describe the possible states of the system. The idea then is that $\mathcal{H}$ doesn't necessarily is a space of functions.
Indeed, Cohen defines (or doesn't define) $\mathcal{H}$ as the space of kets $|\psi\rangle\in \mathcal{H}$, being the kets just vectors encoding the states of the system.
The second postulate states that for each physically observable quantity there is associated one hermitian operator $A$ such that the only possible values to be measured are the eigenvalues of $A$ and such that
If $A$ has discrete spectrum $\{|\psi_n\rangle : n \in \mathbb{N}\}$ then the probability of measuring the eigenvalue $a_n$ on the state $|\psi\rangle$ is $\langle \psi_n | \psi\rangle$ considering that $|\psi\rangle$ is normalized.
If $A$ has continuous spectrum $\{|\psi_{\lambda}\rangle : \lambda \in \Lambda\}$ then the probability density on state $|\psi\rangle$ for the possible eigenvalues is $\lambda \mapsto \langle \psi_\lambda | \psi\rangle$
If, for example, the position operator $X$ for particle in one-dimension, exists, and if its eigenvectors are $|x\rangle$ with eigenvalues $x$, for each $x\in \mathbb{R}$, the probability density of position is $\langle x |\psi\rangle$ which is a function $\mathbb{R}\to \mathbb{C}$ and we recover the wavefunction.
This formulation, though, seems to be more general. In that case, wavefunction is just the information about one possible kind of measurement which we can obtain from the postulates. There is nothing special with it.
Now, although quite elegant and simple, this is not even a little rigorous. For example: the position operator hasn't been defined! It is just "the operator associated to position with continuous spectrum", but this doesn't define the operator. On the book, it is defined on the basis $\{|x\rangle\}$, but this set is defined in terms of it, so we get circular.
Another problem is that usually we are dealing with unbounded operators which are not defined on the whole of $\mathcal{H}$. And an even greater problem is that $\mathcal{H}$ was never defined!
I've been looking forward to find out how to make this rigorous, but couldn't find anything useful. Many people simply say that the right way is to consider always $L^2(\mathbb{R}^3)$, so that all of this talk is nonsense. But I disagree, I find it quite natural to consider this generalized version.
The only thing I've found was the idea of rigged Hilbert spaces, known also as Gel'fand triple. I've found not much material about it, but anyway, I didn't understand how it can be used to make this rigorous.
In that case, how does one make this idea of space of states, or space of kets, fully rigorous, overcoming the problems I found out, and possibly any others that may exist? Is it through the Gel'fand triple? If so, how is it done?