I want to find discrete-time Markov process (its stochastic transition matrix) equivalent to a given contiouse time Markov process.

I googled it, what I found is that if we denote $q_{ij}$ as the transition rate in the Markov process to go from state $i$ to state $j$, then the transition probability to transit from $i$ to $j$ in the discrete version would be $p_{ij} = \frac{q_{ij}}{v_i}$, where $v_i = \sum_{k}{q_{i k}}$.

Okay, based on this, we will get a stochastic transition matrix in which the diagonal elements are zero. But the point is base don this transition matrix in each time step we will see the transition occurs, while in general transition matrix, the diagonal elements are not necessarily zero which means this is probable that transition does not occur is some time steps.

Even if we compare this embedded discrete time Markov with the continuous one, in contentious one we have holdings times which have an exponential distribution, resulting in that the total number of transition in a fix time duration is not equal. So I do not think embedded discrete-time Markov chain obtained from this recipe is equivalent to the original continuous time. Am I right?

Could you please let me know what is the standard way to go from a continuous time process to its equivalent discrete-time Markov chain?

Thank you so much in advance!

  • 1
    $\begingroup$ Indeed this is a very bad approximation of the continuous Markov process since one does not distinguish between a given set $(q_{ij})_j$ of transition rates from $i$, and any of its multiples. A more sensible approximation would be $$p_{ij}=aq_{ij}\qquad p_{ii}=1-a\sum_{j\ne i}p_{ij}$$ for $a>0$ small enough. Or the exact solution $$p_{ij}=(e^{bQ})_{ij}$$ which corresponds to observing the process $(\xi_t)$ at regularly spaced times, that is, $$X_n=\xi_{nb}$$ $\endgroup$
    – Did
    Oct 5, 2017 at 15:01
  • $\begingroup$ Oh, and by the way, the suggestion in your post corresponds to enumerating the sequence of different states visited by the Markov process $(\xi_t)$, that is, $X_n=\xi_{\tau(n)}$ with $\tau(0)=0$ and $$\tau(n+1)=\inf\{t>\tau(n)\mid\xi_t\ne\xi_{\tau(n)}\}$$ $\endgroup$
    – Did
    Oct 5, 2017 at 15:04
  • $\begingroup$ Thanks for your comments. What is $b$? I know this exponential relationship in the form $P(t) = e^{G t}$. But, I want to implement the embedded Markov chain in simulations... So I do not think the exponential form is useful in this situation, is not it? $\endgroup$
    – Soom
    Oct 6, 2017 at 0:21


You must log in to answer this question.