Is it possible to find a Discrete Time Markov Chain, with exactly three stationary distributions $\pi_1, \ \pi_2, \ \pi_3$? For a finite chain, i suppose it doesn't work, because it's possible to find either one or infinite stationary distributions. But, for infinite case? Unfortunately, i have no proof for this.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Little bit general theory is required to give a proper and understandable answer. So, here goes.
For a Markov chain on a countable state space ${\cal S}$, we say a state $i $ leads to $j$ ($ i \rightsquigarrow j $) if $ p_{ij}^{(n)} > 0 $ for some $ n \geq 0 $. We say $i$ communicates with $j$ ($ i \leftrightsquigarrow j$) if $ i $ leads to $ j $ and $ j $ leads to $ i$, i.e., $ i \rightsquigarrow j, j \rightsquigarrow i $.
It is easy to check that that $ i $ communicates with $j$ is a equivalence relation on the state space. So, we may split the state space into union of disjoint equivalence classes, which we call communicating classes, i.e., in a communicating class, every state will lead to every other state.
The communicating class property is very important in the sense that all states in the same communicating class will behave in the same way, i.e., if one state is transient / null recurrent / positive recurrent, all states will be transient / null recurrent / positive recurrent respective. Furthermore all states will also have the same period.
Next, we introduce another concept. A class of states is called closed if any state state in the class do not lead to any state outside the class, i.e., if $ {\cal C} $ is a closed class then $ p_{i,j}^{(n)} = 0 $ for all $ n \geq 0 $ for any $ i \in {\cal C}, j \not\in {\cal C} $. The importance of the closed property is that when the Markov chain enters into the set, it can't go out of it, i.e., if $ X_0 \in {\cal C}$, then $ X_{n} \in {\cal C}$ for all $ n \geq 1$. Thus, the Markov chain restricted to the closed class it itself be a Markov chain. More precisely, if we construct a new matrix from the transition matrix keeping only the entries corresponding to the states of the closed class ${\cal C}$, the new matrix will turn out to be a transition matrix and hence will give rise to a Markov chain with state as $ {\cal C}$ (instead of ${\cal S}$).
Recurrence of a state has a strong connection with leading property. It is not very difficult to show that if $ i $ is recurrent and $ i \rightsquigarrow j $, then it must be the case that $ j \rightsquigarrow i $, i.e., $ i \leftrightsquigarrow j $.
A consequence of this will be that a recurrent communicating class (i.e., containing at least one, and hence all, recurrent state) must be closed. Indeed, if a recurrent class $ {\cal C} $ not closed, then there is $ i \in {\cal C }, j \notin C$ such that $ i \rightsquigarrow j $. But we know that since $ i $ is recurrent, we must have $ j \rightsquigarrow i $ so that $ i \leftrightsquigarrow j$, i.e., $ j \in {\cal C}$ which is a contradiction.
From all this above, we may write that the state space as follows: \begin{equation*} {\cal S} = \cup_{ i } {\cal C}_i \cup T \end{equation*} where each $ C_i $ is an enumeration of positive recurrent communicating (closed) classes and $ {\cal C}_i \cap {\cal C}_j = \emptyset $ for $ i \neq j$, $ T $ is the union of all remaining classes (containing only null recurrent states and transient states).
Few words about this decomposition. There is no necessity that both will exist. For example, simple symmetric random walk in one dimension has only null recurrent states, but if it is asymmetric ($ p \neq 1/2 $) then there are only transient states whereas for reflective random walk on $ \{ 0, 1,2 \dotsc, \} $ with boundary at $ 0 $ and $ p < 1/2 $ has only one positive recurrent class, namely the whole state space. Only thing that we can say, if the state space is finite there are at least one recurrent state and no null recurrent states, i.e., every recurrent state must must be positive recurrent. There is also no restriction on the number of positive recurrent classes. For example, in Gambler's ruin problem with total capital $ N $, both $ \{ 0 \} $ and $ \{ N \} $ are positive recurrent (both absorbing states) and remaining states are transient.
A quick result about the stationary distribution for a chain
If $ \pi $ is a stationary distribution of the chain, then \begin{equation*} \pi (x) = 0 \text{ for all } x \text{ null recurrent or transient.} \end{equation*}
One consequence is that if the Markov chain does not have any positive recurrent state, then it does not admit a stationary distribution.
The main result is :
Any irreducible positive recurrent chain has a unique stationary distribution.
Hence, for each positive recurrent communicating class, we may consider the chain restricted to the class. Easy to observe that this restricted Markov chain will be irreducible positive recurrent and hence will have a unique stationary distribution from the above result. For a positive recurrent communicating class $ {\cal C}$, let us call the stationary distribution on ${\cal C} $ by $ \tilde{\pi_{\cal C}} $. The stationary distribution $ \tilde{\pi_{\cal C}} $ now may be extended to a distribution for the whole state space by setting the probabilities of all states outside the class to $0$, i.e., \begin{equation*} \pi_{\cal C} (x) = \begin{cases} \tilde{\pi_{\cal C}} (x) & \text{ if } x \in {\cal C} \\ 0 & \text{ if } x \notin {\cal C}. \end{cases} \end{equation*} It is easy to check that the extended distribution $ \pi_{\cal C} (x) $ is indeed a stationary distribution of the whole Markov chain (we just need to check that $ \pi_{\cal C} $ satisfy the balance equations).
Thus, for each positive recurrent communicating class $ {\cal C}$, we obtain a stationary distribution $ \pi_{\cal C} $ on the whole Markov chain. These are distinct as by construction, the support of $ \pi_{\cal C} $ is $ {\cal C} $ and two distinct communicating classes are disjoint.
Observe that any convex combination of these stationary distributions $ \{ \pi_{\cal C} : {\cal C} $ is positive recurrent communicating class$\}$ is also stationary distribution of the Markov chain (each of these satisfy the balance equation and hence the convex combinations).
To illustrate this, consider the Gambler's ruin problem where $ \{ 0 \} $ and $ \{ N \} $ are the only positive recurrent communicating classes. Thus, we have $ \pi_{\{ 0\}} $ and $ \pi_{\{ N\}} $ are two stationary distributions on the whole state space $ \{ 0,1, \dotsc, N \} $ where \begin{equation*} \pi_{\{ 0\}} (i) = \begin{cases} 1 & \text{ if } i = 0 \\ 0 & \text{ if } i \neq 0; \end{cases} \quad \text{ and } \pi_{\{ N\}} (i) = \begin{cases} 1 & \text{ if } i = N \\ 0 & \text{ if } i \neq N. \end{cases} \end{equation*} Thus, for any $ \lambda \in [0,1] $, the probability distribution $ \lambda \pi_{\{ 0\}} + (1-\lambda) \pi_{\{ N\}} $ is also a stationary distribution of the chain.
We may ask whether these are all the stationary distributions. The answer is yes. If $ \pi $ is a stationary distribution, then it may be expressed as a convex combination of extended stationary distributions as defined above.
So, the upshot of all this is the following: the Markov chain will have no stationary distribution if it has no positive recurrent communicating class (not possible in finite space space case). It will have a unique stationary distribution if it has only one positive recurrent communicating class. If it has more than one positive recurrent communicating classes, there are uncountably many stationary distributions. In this case, stationary distributions are given by the convex hull of extended stationary distributions obtained from each positive recurrent communicating class.