0
$\begingroup$

I have 2 questions about continuous-time Markov Chain.

Let's consider the following example.

Be $\{X(t),t\geq 0\}$ a continuous time Markov chain with state space $S_X=\{1, 2, 3\}$ and the following transition matrix \begin{equation*} Q = \begin{pmatrix} -4& 2 & 2\\1.5 & -2.5 & 1 \\ 0&1&-1\end{pmatrix}\end{equation*} Finally, we have $X(0)=3$.

I would like to know how to compute 2 differents probabilities.

First, the probability that the chain is in the state 1 just after the second jump. About this one, I've noticed that the only path from 3 to 1 is to go from 3 to 2 to 1. However, I don't have any idea about how to compute this.

Also, the probability that the chain stays in state 3 until time t=1. I have no idea how to do this. Cause you give me some hint ?

Thanking you all in advance !

N.B. : I never had any courses about continuous time markov chain, and I found the books about this subject not helpful, so sorry if my questions are a little dumb.

$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

A useful concept in the analysis of time-homogeneous CTMCs is the decomposition into the jump chain and holding times. The jump chain is a DTMC (discrete time Markov chain) with transition probability $p_{ij}=\frac{q_{ij}}{-q_{ii}}$ for $j \neq i$, $p_{ii}=0$. The holding times are independent exponential random variables with rate parameter $-q_{ii}$ (where $i$ is the state that you are currently waiting at).

As for what these actually do, the value of the jump chain at the discrete time $n$ tells you where you are at after $n$ jumps have occurred. Knowing the values of the jump chain, the holding times tell you how long you wait between jumps.

The jump chain answers your first question. Technically what you're asked to get is an entry of the matrix $P^2$. But as you pointed out, you're looking at two jumps, you're starting at 3 and the first jump from $3$ is always to $2$. So you can really just read the answer off from $Q$ without any matrix multiplication.

The holding times answer your second question.

$\endgroup$
7
  • $\begingroup$ Thanks for your answer ! If I understand well, the first answer should be $\displaystyle\frac{3}{5}$. For the second one, it means that the time I stay in state 3 follows an exponential distribution with parameter 1, so the probability that after 1 minute I'm still in state 3 is $e^{-q_{33}*1} = e^{-1}$ ? Thanks again for your help $\endgroup$
    – Arthur Boivert
    Oct 14, 2021 at 11:56
  • $\begingroup$ @ArthurBoivert That's right. $\endgroup$
    – Ian
    Oct 14, 2021 at 12:55
  • $\begingroup$ thanks for your help !! $\endgroup$
    – Arthur Boivert
    Oct 14, 2021 at 12:57
  • $\begingroup$ I just have one dumb question, for my second question how do you write the probability mathematically ? Is $P(X(1)=3\vert X(0)=3)=1-F_{T_3}(1) = e^{-1}$ where $T_3$ is the holding time for state 3 is correct ? $\endgroup$
    – Arthur Boivert
    Oct 15, 2021 at 9:17
  • $\begingroup$ @ArthurBoivert The LHS the probability that you're at $3$ at time $1$ given that you started there, regardless of what steps you took along the way. The RHS is the probability that you never left. It's possible, albeit somewhat unlikely in this particular case, that you did several jumps along the way, so the LHS is somewhat larger. $\endgroup$
    – Ian
    Oct 15, 2021 at 13:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .