# Probability of being at given state in a continuous time Markov chain

Question:

Consider a two-state continuous time Markov chain (with states $$1$$ and $$2$$) in which the holding rate at state $$1$$ is $$\lambda_1=2$$, and the holding rate at state $$2$$ is $$\lambda_2=3$$.

Suppose that we start at state $$1$$ (i.e. $$X_0 = 1$$). Find the probability that $$X_t = 1$$ as $$t \rightarrow \infty$$.

Attempt:

So the transition matrix for the underlying discrete Markov chain is

$$\mathbf P = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}$$

whereas the $$Q$$-matrix for the continuous time Markov chain is

$$\mathbf Q = \begin{pmatrix} -2 & 2 \\ 3 & -3 \end{pmatrix}$$

Let $$\xi = (\xi_1 , \xi_2)$$ be the stationary distribution. At this point, do I solve $$\xi \mathbf P = \xi$$ or $$\xi \mathbf Q = 0$$ to find the stationary distribution?

And, after I have found the stationary distribution, is the required probability simply $$\xi_1$$?

Any hints/suggestions would be much appreciated. Thanks!

• You would solve $\xi Q = 0$, and yes, the probability in question is $\xi_1$. Apr 24, 2019 at 19:02

First we can do some reasoning about why we use $$\pi Q = 0$$ to get the stationary distribution. According to the definition of stationary distribution, the equation we are to solve is $$\pi P(t)=\pi$$, where $$P(t)$$ is the transition matrix of the process.

One thing to note is that the $$\mathbf{P}$$ you mentioned is not the same as $$P(t)$$, where the latter is a function of $$t$$. $$\mathbf{P}$$ you mentioned is the jump matrix. It is kind of like what you said, an underlying discrete-time transition matrix.

Let's get back to the equation $$\pi P(t)=\pi$$. Obviously it is not easy to solve. Hence we can take derivative with respect to $$t$$, and the LHS becomes $$\pi Q$$, and the RHS becomes $$0$$.

To answer your second question. The question asked explicitly for $$\mathbb{P}(X(t)=1)$$ as $$t\rightarrow \infty$$. Hence the distribution we want is actually the limiting distribution. Under the condition that $$X$$ is irreducible with a standard semigroup $$\{\mathbb{P}(t),t\geq 0\}$$ of transition probabilities, we can say that the stationary distribution is also the limiting distribution.

Also, for this question the full balance equation $$\pi Q = 0$$ is also the detailed balance equation, indicating the process is reversible.

You seem to want to use row vectors where most people work with column vectors for the distribution vector. In terms of the distribution $$\vec{\xi}(t) = \pmatrix{\xi_1(t)\\ \xi_2(t)}$$ the matrix describing the evolution of the system is $$\frac{d\vec{\xi}}{dt} = \pmatrix{-2 & 3\\ 2 & -3} \vec{\xi}$$ You are trying for a stationary state, in which $$\frac{d\vec{\xi}}{dt} = 0$$ so you want to solve the simultaneous equations $$\pmatrix{-2 & 3\\ 2 & -3} \pmatrix{\xi_1\\ \xi_2} = 0 \\ \xi_1+\xi_2 = 1$$ and you answer for the probability of state $$1$$ is indeed $$\xi_1$$ per that solution.

Note that although there appear to be $$3$$ equations in $$2$$ unknowns, the first two equations are not independant so there will be a solution.