6
$\begingroup$

Let their be a Markov chain on state space $S = \{0,1,2,...\}$. Transition probabilities are given as:

$p_{0,1} = 1$, $p_{i,i+1} = p$, $p_{i+1, i} = q$

where $0<p,q<1$ and $p+q=1$. For $p<q$ , I need to find the stationary distribution (say $\pi$) of this Markov chain.

I got the balance equations: $$\pi_{0} =q.\pi_{1}$$ $$\pi_{i} = p.\pi_{i-1}+ q.\pi_{i+1}$$ and normalization conditoin: $$\sum_{i=0}^{\infty}\pi_{i} = 1$$

Using these relations, how can I get the stationary distribution?

$\endgroup$
  • 1
    $\begingroup$ Check out, that you can write balance equations between only two neighboring nodes, so things become much more easier $\endgroup$ – dEmigOd Feb 19 '18 at 6:49
7
$\begingroup$

The condition you give is the correctly identified global balance equation for $\pi, \, P$, however it is much easier to find the closed form of the stationary distribution by proving that it satisfies the detailed balance condition

$$\pi(i) P(i,j) = \pi(j)P(j,i), \qquad \forall \, i,j\, \in S.$$

If we can find such a probability vector $\pi$ which satisfies this condition, then $\pi$ is known to be a stationary distribution for $P$.

In the context of your Markov chain, we know that $P(i,j) = 0$ if $j \neq i \pm 1$ and it suffices to find $\pi$ satisfying $$ \pi(i)P(i,i+1) = \pi(i+1)P(i+1,i)$$ That is $$\pi(i)p = \pi(i+1)(1-p).$$ If we write $\rho = p/(1-p)$ then we have the recurrence relation $$\pi(i+1) = \rho \pi(i) = \cdots = \rho^{i} \pi(1) = \frac1p \, \rho^{i+1} \pi(0),$$ where in the final equality we use that the detailed balance at $i = 0$ is given by $\pi(0) = (1-p) \pi(1)$.

Since the stationary distribution must sum to $1$ (it is a probability distribution) we have

$$1 = \sum_{i=0}^{\infty} \pi(i) = \frac{\pi(0)}p \sum_{i=0}^\infty \rho^i$$ You can now use the geometric series identity to identify when this is well defined, and to find the exact formula for the stationary distribution.

Note: In the above I worked with the method of assuming detailed balance to be true, and then showing that this was in fact the case (for certain $p$, which you will need to check). An alternate approach would have been to first demonstrate that the process is reversible by proving Kolmogorov's Criterion (which shows reversibility, which is equivalent to detailed balance), and then deriving the detailed balance relationship after.

$\endgroup$
4
$\begingroup$

Just write those equations between only two nodes, like you have for $0$. $$1 \cdot \pi_0 = q \cdot \pi_1$$ $$p \cdot \pi_1 = q \cdot \pi_2$$ $$ \ldots $$ $$\pi_{k+1} = \frac{p}{q} \cdot \pi_k = \left(\frac{p}{q}\right)^{k}\cdot \pi_1$$

Then you have $$1 = \sum\limits_{k=0}^{\infty}\pi_k = q \cdot \pi_1 + \pi_1 + \left(\frac{p}{q}\right)^{1}\cdot \pi_1 + \left(\frac{p}{q}\right)^{2}\cdot \pi_1 + \ldots$$

$$1 = \pi_1 \cdot \left( q + 1 + \left(\frac{p}{q}\right)^{1} + \left(\frac{p}{q}\right)^{2} + \ldots\right)$$

$$\pi_1 = \frac{1}{\frac{1}{1 - p/q} + q}$$

you need to be careful, so $p < q$, otherwise no stationary distribution exists.

$\endgroup$
  • $\begingroup$ Why do you think it is obvious that you can write the relationship $p \pi_1 = q \pi_2$? This solution fails to mention that this relationship is a very special case of the detailed balance equation, and further that MCs need to be proven to satisfy detailed balance, as this is not true of all MCs (as in your example, $p \geq q$). Having a stationary distribution is not a sufficient condition for detailed balance; consider for instance the process with transition matrix $P = \left( \begin{array}{ccc} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & 0 & 0 \end{array}\right).$ $\endgroup$ – owen88 Feb 19 '18 at 7:04
  • $\begingroup$ What's your point? $\pi = (\frac{1}{3}, \frac{1}{3}, \frac{1}{3})$ is stationary distribution, it is though unachievable, if not started at $\pi$, this is because the MC is periodic. But this is the case in OP's question also ! Stationary distribution is the eigen-vector of $\lambda=1$. $\endgroup$ – dEmigOd Feb 19 '18 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.