# Probability of being at given state in a continuous time Markov chain

Question:

Consider a two-state continuous time Markov chain (with states $$1$$ and $$2$$) in which the holding rate at state $$1$$ is $$\lambda_1=2$$, and the holding rate at state $$2$$ is $$\lambda_2=3$$.

Suppose that we start at state $$1$$ (i.e. $$X_0 = 1$$). Find the probability that $$X_t = 1$$ as $$t \rightarrow \infty$$.

Attempt:

So the transition matrix for the underlying discrete Markov chain is

$$\mathbf P = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}$$

whereas the $$Q$$-matrix for the continuous time Markov chain is

$$\mathbf Q = \begin{pmatrix} -2 & 2 \\ 3 & -3 \end{pmatrix}$$

Let $$\xi = (\xi_1 , \xi_2)$$ be the stationary distribution. At this point, do I solve $$\xi \mathbf P = \xi$$ or $$\xi \mathbf Q = 0$$ to find the stationary distribution?

And, after I have found the stationary distribution, is the required probability simply $$\xi_1$$?

Any hints/suggestions would be much appreciated. Thanks!

• You would solve $\xi Q = 0$, and yes, the probability in question is $\xi_1$. – Math1000 Apr 24 at 19:02

You seem to want to use row vectors where most people work with column vectors for the distribution vector. In terms of the distribution $$\vec{\xi}(t) = \pmatrix{\xi_1(t)\\ \xi_2(t)}$$ the matrix describing the evolution of the system is $$\frac{d\vec{\xi}}{dt} = \pmatrix{-2 & 3\\ 2 & -3} \vec{\xi}$$ You are trying for a stationary state, in which $$\frac{d\vec{\xi}}{dt} = 0$$ so you want to solve the simultaneous equations $$\pmatrix{-2 & 3\\ 2 & -3} \pmatrix{\xi_1\\ \xi_2} = 0 \\ \xi_1+\xi_2 = 1$$ and you answer for the probability of state $$1$$ is indeed $$\xi_1$$ per that solution.

Note that although there appear to be $$3$$ equations in $$2$$ unknowns, the first two equations are not independant so there will be a solution.

First we can do some reasoning about why we use $$\pi Q = 0$$ to get the stationary distribution. According to the definition of stationary distribution, the equation we are to solve is $$\pi P(t)=\pi$$, where $$P(t)$$ is the transition matrix of the process.

One thing to note is that the $$\mathbf{P}$$ you mentioned is not the same as $$P(t)$$, where the latter is a function of $$t$$. $$\mathbf{P}$$ you mentioned is the jump matrix. It is kind of like what you said, an underlying discrete-time transition matrix.

Let's get back to the equation $$\pi P(t)=\pi$$. Obviously it is not easy to solve. Hence we can take derivative with respect to $$t$$, and the LHS becomes $$\pi Q$$, and the RHS becomes $$0$$.

To answer your second question. The question asked explicitly for $$\mathbb{P}(X(t)=1)$$ as $$t\rightarrow \infty$$. Hence the distribution we want is actually the limiting distribution. Under the condition that $$X$$ is irreducible with a standard semigroup $$\{\mathbb{P}(t),t\geq 0\}$$ of transition probabilities, we can say that the stationary distribution is also the limiting distribution.

Also, for this question the full balance equation $$\pi Q = 0$$ is also the detailed balance equation, indicating the process is reversible.