# First return times and continuous markov chains.

We are given a generator matrix $Q$ (Q-matrix) for a continuous time Markov chain $(X_t)_t$. We want to calculate the probabilities of:

• returning to State 3 before State 1, while starting at State 3: $P(T_3 < T_1 | X_0=3)$
• returning to State 3 before State 1, while starting at State 2: $P(T_3 < T_1 | X_0=2)$
• The following expected time: $E(\min\{T_1, T_3\} | X_0 = 2)$

In the above, $T=\inf\{t>0; X_t=i\}$

Here is $Q$:

$\begin{pmatrix} -6 & 3 & 1 & 2\\ 3 & -8 & 3 & 2\\ 2 & 1 & -7 & 4\\ 1 & 1 & 3 & -5 \end{pmatrix}$

I would be so grateful for your help!

I know it's similar to the "Gambler's ruin" problem, which I can solve, but I'm not sure if this can be solved in a similar fashion. The problem is that here, we start in State 3 but then want to calculate the probability of returning back to 3 before hitting 1.

• When the starting point is "inside" the domain, there is a very standard approach. Specifically, to find the probability of hitting $A$ before $B$, you solve $Q(x)=0$ with the boundary conditions $x(A)=1$ and $x(B)=0$. Then $x(2)$ answers your second question. Similarly, to find the expected time to hit $\{ 1,3 \}$ from $2$, you compute $x(2)$ where $Q(x)=-1$ with the boundary condition $x(\{ 1,3 \})=0$. This answers your third question. The trickier matter is when you start out "on the boundary", as in your first case. – Ian Jan 30 '16 at 23:15
• Thank you! Yes, exactly, the first case is the main issue for me as I can't figure out how to get a start on it. – feli Jan 30 '16 at 23:32
• Well, you know what you would do if you started at 1, 2, or 4 and needed to hit 3 before 1. Add together those results, weighted by the probability of transitioning to that starting state from state 3 in the first step. Also, I think your formula in the first case is not exactly correct, in that case I think you want $T$ to be after the first transition from state $3$. As written your first quantity will always be $1$. – Ian Jan 31 '16 at 0:16
• For the first question: my problem is that I don't know how to take into account the "time" parameter. I guess I can write $P(T_3 < T_1 | X_0=3) = P(T_3 < T_1 | X_0=2)P(J_1=2) + P(T_3 < T_1 | X_0=4)P(J_1=4) + 0$, where $J_1$ denotes the first jump. We know that the first jump equals the first "holding time", hence, it is exponentially distributed with parameter $q_3 = 7$. But do we even need that? Can we just say that $P(J_1=2) = \frac{q_{32}}{q_3}=\frac{1}{7}$? – feli Jan 31 '16 at 10:36
• The holding time has nothing to do with anything, what matters is the distribution of the process after the first jump. And that is indeed $P(j)=\frac{q_{ij}}{q_i}$ (assuming you mean $q_i=-q_{ii}$ and that $q_i \neq 0$). – Ian Jan 31 '16 at 14:47