What is the mathematical meaning of a quantum operator? (Context: I am learning functional analysis using the book by Erwin Kreyszig "Introductory functional analysis with applications." The last chapter is dedicated to the applications of functional analysis in quantum mechanics. (It's pp 572-586.).)
Suppose we have a ``state'' defined as a function in $\psi(q) \in L^2[-\infty, \infty]$ such that $||\psi||=1$.
We can then find the mean value of $q$ as $\int_{-\infty}^\infty q \psi^*(q)\psi(q)dq$. {Formula (3), page 574} This perfectly makes sense since $q$ is a scalar, $\psi^*(q)\psi(q)$ is also a scalar ( for every value of $q$). So, my random variable is $q\sim \psi^*(q)\psi(q)$, and I am taking an expectation. 
I could find a mean value of many other functions of my $q$. Say, $q^2$ or $e^{-q^2}$. I could do it by two methods -- I could take an integral directly $\int_{-\infty}^\infty f(q) \psi^*(q)\psi(q)dq$, or I could find a distribution of $g = f(q)$ being $g \sim  \mu(f^{-1}(g)) =\phi^*(g)\phi(g)=  \int_{q=f^{-1}(g)}\psi^*(q)\psi(q)dq$ and then take $\int_{-\infty}^\infty g \phi^*(g)\phi(g)dg$. Am I correct up to now?
Then we have the same integral expressed as a dot product:
$R[Q] = <Q\psi, \psi> = \int_{-\infty}^\infty Q \psi^*(q)\psi(q)dq$
{Formula (6) in the book.} In this formula we have replaced $q$ with the operator $Q$, which works by multiplying the input by $q$. The formula still corresponds to the same numbers, but let's note that now it actually should be read as $\int_{-\infty}^\infty [Q \psi(\alpha)](q)\psi^*(q)dq$. This thing, while still being correct, bothers me, because now we don't have clearly distinguishable random variable and its distribution under the integral. 
And then there is the next step: the authors say that you can in the same manner get "mean values" for any operator $T$, say:
$\int_{-\infty}^\infty [T \psi(\alpha)](q)\psi^*(q)dq$
Now for me it doesn't make sense. This is the bit I don't understand.
Firstly, we lose distributivity:
$\int_{-\infty}^\infty [T \psi(\alpha)](q)\psi^*(q)dq \neq \int_{-\infty}^\infty T \cdot |\psi(q)\psi^*(q)|dq$. (On the left we have a scalar, on the right we have an operator.
Secondly, I don't understand how a "mean value" consists of a dot product of two seemingly unrelated function-vectors.
If we could say that every operator T acts like $[T\psi(\alpha)](x) = f(x)\cdot\psi(x)$, where $f(x)$ may depend on $\psi$ but not on $x$, it would be fine since we would be computing $\int_{-\infty}^\infty f(q)\psi(q)\psi^*(q)dq$, but why would such a guarantee be true? 
UDP1:
Now when I think of it, it seems to me a bit more sensible. I.e. we can write 
$[T\psi](x)= [\frac{[T\psi]}{\psi}](x) \psi(x) = f(x)\cdot\psi(x)$
The division is not a very well-behaving thing in general, but since both the enumerator and the denominator here seem to be "well-behaved" function, it should work, right?
Then $f(x) = [\frac{T\psi}{\psi}](x)$ will be my random variable...
Does it even make sense?
Does $f(x)$ have any reasonable name?
 A: If $\psi$ is a state vector, meaning $\|\psi\|=1$, then the expected value of an observable $A$ for the system in the state $\psi$ is
$$
         (A\psi,\psi).
$$
That's what you're trying to show. One must assume that $\psi$ is in the domain of the observable $A$.
A selfadjoint operator $A : \mathcal{D}(A)\subseteq H\rightarrow H$ on a Hilbert space $H$ can be decomposed using the spectral theorem for selfadjoint operators $A$:
$$
              A\psi = \int_{-\infty}^{\infty}\lambda dE(\lambda)\psi,
$$
and $\psi\in\mathcal{D}(A)$ iff
$$
               \int_{-\infty}^{\infty}\lambda^2 d\|E(\lambda)\psi\|^2 < \infty.
$$
The measure $d\|E(\lambda)\psi\|^2$ is a probability measure that is (axiomatically) the probability distribution of $\psi$ with respect to the spectrum of $A$. The expected value of $A$ for the system in state $\psi$ is, therefore,
$$
               \langle A \rangle = \int_{-\infty}^{\infty}\lambda d\|E(\lambda)\psi\|^2 = (A\psi,\psi).
$$
The notation used here is John von Neumann's, who formulated the Spectral Theorem for unbounded selfadjoint operators in his study of Quantum Mechanics around 1930. Unlike the Dirac version, von Neumann's decomposition is generally applicable to bounded and unbounded selfadjoint operators on a Hilbert space, and is rigorously justified. (He used $E$, $\lambda$ in his original work to suggest its primary use in Quantum.)
UPD from the OP (as a comment to the answer):
I think this answer is correct, but I would like to express it in simpler words:
$$(\hat{A} \psi (x), \psi^* (x)) = \int_{-\infty}^{\infty} [\hat{A} \psi] (x) \cdot \psi^*(x)dx = \int_{-\infty}^{\infty} \left(\int_{-\infty}^{\infty}\lambda [\hat{E_\lambda} \psi](x) d\lambda \right)  \cdot \psi^*(x)dx = \int_{-\infty}^{\infty} \left(\int_{-\infty}^{\infty}\lambda \varphi(\lambda) \varphi_\lambda(x) d\lambda \right)  \cdot \psi^*(x)dx$$
Here, $\hat{E_\lambda}$ is a projection operator on an eigenfunction corresponding to $\lambda$ eigenvalue.
Here, $\varphi(\lambda)$ is a an expansion proportionality factor. (The new wavefunction.)
Here, $\varphi_\lambda(x)$ is an eigenfunction in the original basis.
Since the outer integral does not depend on $\lambda$, we can change the order of integration.
$$  \int_{-\infty}^{\infty}\lambda \varphi(\lambda) \left( \int_{-\infty}^{\infty} \varphi_\lambda(x) \cdot \psi^*(x)dx \right) d\lambda   $$
Now the unobvious thing here is that $ \int_{-\infty}^{\infty} \varphi_\lambda(x) \cdot \psi^*(x)dx = \varphi^*(\lambda)$, therefore:
$$ (\hat{A} \psi (x), \psi^* (x)) = \int_{-\infty}^{\infty}\lambda \varphi(\lambda) \varphi^*(\lambda) d\lambda$$
