How are Tr(AB) results restricted? In quantum mechanics one postulates that for each state $i$ there is a matrix $A_i$ and for each measureable dynamical variable $j$ (velocity etc.) there is a matrix $B_j$. Both are Hermitian matrices over complex numbers.
The experimental average of the dynamical variable B is postulated to be
$\text{'average of variable j with state i'}=\operatorname{Tr}(AB)$
(additionally some restrictions are placed on the state density matrix $A\geq 0$ and $\operatorname{Tr}A=1$)
Does someone have an idea how this postulates restricts possible outcomes for the average values? Or can completely general systems from probability theory always be stated with this trace notation? Are there mathematically implicit correlations between different states or variables due to this postulate? Is there a similar framework in probability theory unrelated to QM?
All these questions are meant to be purely mathematically derived from the form of the equation.
 A: The direct analogue in classical probability is that $A$ becomes an arbitrary probability measure $\mu$, and $B$ some random variable $X$. The trace operator is then the expectation value of $X$ relative to measure $\mu$. 
That the two coincide is easily seen in the case where your state space is finite dimensional, so $A$ is actually a matrix. The underlying probability space is a finite set $\Omega$ on which we take the sigma algebra $2^X$. Given an arbitrary probability measure $\mu$ on $\Omega$ (which we identify with a function on the finite set) you can extend it to a probability measure on $\Omega^2$ by setting 
$$ \tilde{\mu}(x,y) = \mu(x) \delta(x,y) $$
This naturally can be seen also as the diagonal matrix corresponding to the Hermitian matrix $A$. 
Then a random variable on $\Omega^2$ is just a scalar function over $\Omega^2$, which we can easily identify with a matrix $B$, and the expectation value can be computed to be the same as the trace of $AB$. 
If you want to encode finite probability spaces in the "quantum" language, just restrict your random variables to those defined on $\Omega$ (so you are in fact looking $\Omega$ included in $\Omega^2$ as the diagonal subset). If you want to encode finite-dimensional quantum mechanics as probability, then after diagonalising $A$ you can extract $\mu$, and in this basis $B$ is just some matrix on $\Omega^2$ that you can identify with $X$. 
For the infinite dimensional case some more care is needed: the main idea is to go through the Borel functional calculus; see also this Wikipedia entry on spectral measure. 
