Often Markov process are linked to automata with transition probilities, and I am looking for ways to define them to make this intuition more explicit in the definition using a definition of automata. For this I am looking for a way to relate the usual definition with the definition of finite state machines.

I came up with a definition given by a transition map $g : \mathbb R \times S \to \mathbb S$, where the first parameter in some way codes the input, which is given by a "control process" which accounts for the stochastic inputs (i.e. the result of a random experiment). But is this definition equivalent to the usual one for discrete, finite state markov processes, i.e. are the following two definitions equivalent:

A discrete, finite state Markov process is a stochastic process $X_n, n = 1,2,3,\ldots$ where $X_n : \Omega \to S$ are random variables with finite image $|S| < \infty$ and such that $$ P(X_{n+1} = t_{n+1} | X_n = t_n, \ldots, X_1 = t_1 ) = P(X_{n+1} = t_{n+1} | X_n = t_n ) $$ for all $n$ and $t_1, \ldots, t_{n+1} \in S$.

And the other definition:

A discrete time, finite state Markov process is a stochastic process $X_n$ such that there exists some control process $Z_n : \Omega \to \mathbb R, n = 1,2,\ldots$ and a function $g : R \times S \to S$ such that $X_{n+1} = g(Z_n, X_n)$.

I could also propose a control process $g : \mathcal A \times S \to S$ where $\mathcal A$ is some $\sigma$-algebra from some probability space, highlighting more that the events of some random experiment serve as inputs (and it is easy to see that for finite $\mathcal A$ we can arrange, by collapsing events, that $g(A, s) \ne g(B, s)$ for $A \ne B$, thereby having not two "parallel" transitions going from a state to the same state).

  • $\begingroup$ Yes the definitions are equivalent, provided one assumes that the process $(Z_n)$ is independent. This is explained in most textbooks. $\endgroup$ – Did Dec 1 '16 at 10:15
  • $\begingroup$ @Did Could you please give references? In mine (Probability and Measure, P. Billingsley) it is not mentioned... $\endgroup$ – StefanH Dec 1 '16 at 16:10
  • 1
    $\begingroup$ On the top of my head, exercise 1.1.3 in Norris Markov chains, theorem 2.1 in Brémaud Markov Chains: Gibbs Fields, Monte Carlo Simulation, exercise 3 of section 6.14 in Grimmett and Stirzaker Probability and Random Processes, and theorem 8.1.1 in Rosenthal A First Look at Rigorous Probability Theory. $\endgroup$ – Did Dec 1 '16 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.