System model in probabilistic form The following is a question I have had for quite some time related to systems and control. Problems in control theory typically write the state dynamics as
$$x_{t+1}= f(x_t,u_t,w_t)$$
where $x_t\in\mathcal{X}$ is the state, $u_t\in\mathcal{U}$ is the control, and $w_t$ is a noise term. This "state space" form is often stated as a general model, which includes problems such as MDPs as a special case.
My question relates to how to specifically instantiate the general model to describe an MDP. The state and action spaces are fine, set $\mathcal{X}$ and $\mathcal{U}$ to be finite sets, but it's not clear how to define the function $f$ and the noise terms $w_t$ to capture the dynamics of an MDP.
Question: Recall that the state in an MDP evolves according to a conditional probability distribution $p(x_{t+1}\mid x_t,u_t)$. What is the function $f$ and the distribution on $w_t$'s such that the general model describes an MDP?
 A: Let the noise $W_t$ be a random variable that is (independently of everything) uniformly distributed over the interval $[0, 1]$. Then, the function $f(x_t, u_t, w_t)$ would be given by
$$f(x_t, u_t, w_t)=\begin{cases}1 \hspace{8mm}w_t<P(1|x_t, u_t)\\
2\hspace{8mm}P(1|x_t, u_t)\leq w_t<P(1|x_t, u_t)+P(2|x_t, u_t)\\
\vdots\\
\end{cases}$$
A: If $\ \cal{X}\ $ and $\ \cal{U}\ $ are the state space and action space of the the Markov decision process you're trying to represent, and its transition probabilities are given by
$$
P\big(s_1,s_2,u\big)=Pr\big(x_{t+1}=s_2\,\big|\,x_t=s_1, u_t=u\big)\ ,
$$
then you can take the noise of your "state space" representation to be a sequence $\ w_1, w_2,\dots,w_t,\dots\ $ of independent random variables taking values in the space $\ \cal{W=X^{X\,\times U}}\ $ of functions $\ w:\cal{X\,\times U\rightarrow X}\ $.   If $\ \cal{X}\ $ and $\ \cal{U}\ $ are finite, then so is $\ \cal{W}\ $ and you can take the probability mass function of $\ w_t\ $ to be given by
$$
Pr\big(w_t=w\big)=\prod_{x\in{\cal X}\\ u\in{\cal U}}P(x, w(x,u), u)\ .
$$
If you now take $\ f:\cal{X\times U\times W\rightarrow X}\ $ to be given by
$$
f(x,u,w)=w(x,u)\  
$$
the process $\ x_t\ $ generated by
$$
x_{t+1}=f\big(x_t,u_t,w_t\big)
$$
will be essentially the same stochastic process as that generated by the original Markov decision process.
If $\ \cal{X}\ $ and $\ \cal{U}\ $ are not both finite, then the definition of $\ Pr\ $ given above won't do.  However, if $\ \cal{X}\ $ and $\ \cal{U}\ $ are discrete, you can still construct a probability mass functions on the discrete spaces $\ \cal{W_{AB}=X^{A\,\times B}}\ $, where $\ \cal A\ $ and $\ \cal{B}\ $ are finite subsets of $\ \cal{X}\ $ and $\ \cal{U}\ $,  by
$$
Pr\big(w_t\,_{|\cal A\times B}=w\big)=\prod_{x\in{\cal A}\\ u\in{\cal B}}P(x, w(x,u), u)\ .
$$
I expect these can be parlayed by Kolmogorov extension into a probability measure on a $\sigma$-algebra of subsets of $\ \cal{X^{X\,\times U}}\ $, and with this as the distribution of $\ w_t\ $ and the same $\ f\ $ as above, you will be able to get the same result as in the finite case.  However, I don't have much current experience in constructions of this sort, so I'm not sufficiently motivated to try and work out all the gory details.
