# Definition of Markov chain: question about stationarity

My notes define a Markov chain in the following way:

A Markov chain is a stochastic process $$\{X_n\}_{n=0}^\infty$$ with a state space $$\mathcal{S}$$ that is at most countable and that satisfies the Markov-property:

For every $$n \geq1$$ and $$(x_0, \dots, x_{n+1})\in \mathcal{S}^{n+2}$$ with $$\mathbb{P}(X_0 = x_0, \dots, X_n = x_n) > 0$$ we have

$$\mathbb{P}(X_{n+1} = x_{n+1}\mid X_n = x_n, \dots, X_0 = x_0) = \mathbb{P}(X_{n+1} = x_{n+1}\mid X_n = x_n)$$

Moreover, the probabilities $$\mathbb{P}(X_{n+1} = y \mid X_n = x)$$ are stationary for all $$x, y \in \mathcal{S}$$ with $$\mathbb{P}(X_n = x) > 0$$: they do not depend on $$n$$ (= for all $$n \geq 0$$ the probabilities give the same result).

Question:

My question concerns the stationary part of the definition. Does this definition imply the following equivalence? (to make all conditional probabilities exist)

$$\exists n \geq 0: \mathbb{P}(X_n = x) > 0 \iff \forall n \geq 0: \mathbb{P}(X_n = x) >0$$

• Does your question miss a ">0" in the end? If so, what if you consider a chain with two states (0 and 1) and where at each time the chain moves from 1 to 0 (or from 0 to 1) with probability one. – user52227 Mar 7 at 11:45
• @user52227 I fixed it. Thanks. – user370967 Mar 7 at 12:27

My PhD is in discrete Markov chains, and I've never heard this use of terminology. What you have is a time homogeneous Markov chain. What this means is the following.

When you look at $$P_n = \{ P(X_{n+1} = y \mid X_n = x) \mid x,y \in S\}$$, you are looking at all the possibilities for the transition from the Markov chain's $$n$$-th position to its $$(n+1)$$-st. So first you need to look at $$P_0$$, then $$P_1$$, etc.

If $$P_n$$ is independent of $$n$$, then this means that each jump has the same distribution. Overwhelmingly such time homogeneous Markov chains are studied---although perhaps we, as a community, are wrong to not study inhomogeneous ones more...

The standard use of the term "stationarity" is very different. A distribution $$\pi$$ (on $$S$$) is called a stationary measure (my preferred term is "invariant distribution"; it's also known as an "equilibrium distribution") if the distribution of $$X_n$$ is $$\pi$$ when $$X_0 \sim \pi$$. So if a time-homogeneous chain has matrix $$P$$, then $$\pi P = \pi$$. It's called "equilibrium" distribution, since under certain (fairly weak) conditions one can show that, regardless of $$X_0$$, we have $$P(X_n = j \mid X_0 = i) \to \pi(j)$$ as $$n \to \infty$$.

Note that "$$\exists n : P(X_n = x) > 0$$" certain doesn't not imply that $$P(X_n = x) > 0$$ for all $$n$$. For example, take the simple Markov chain that has two vertices in the state space and always moves to vertex 1: $$P = \begin{pmatrix} 1 & 0 \\ 1 & 0 \end{pmatrix}.$$ If you start at the second vertex, or let $$X_0$$ be some measure that is not a point mass at $$1$$, then the claim is violated: $$P(X_0 = 2) > 0$$ but $$P(X_n = 1) = 1$$ for all $$n \ge 1$$.

An excellent book on Markov chains is by James Norris. It really is the go-to reference, in my opinion. Grimmett also has some good stuff. (Just search for their names and "Markov chains" on your favoured search engine, aka Google.)

• Thanks for your answer. The wikipedia page does mention "stationary markov chains", which is what I meant: en.wikipedia.org/wiki/Markov_chain#Discrete-time_Markov_chain. – user370967 Mar 6 at 20:16
• Also, I'm sorry to inform you that this doesn't really answer my question. – user370967 Mar 6 at 20:17
• I don't understand your question then. Maybe there's a typo in the final line? – Sam T Mar 7 at 9:38
• Do you understand (or agree with) my definition of markov chain? If so, the question I'm asking is that if a transition probability $P(X_{n+1}=y\mid X_n = x)$ exists (i.e. $P(X_n =x)>0$), then the stationarity implies it exists for all $n$, thus does it follow that $P(X_n =x)>0$ for all $n$ in that case? ( and conversely). Sorry that the question isn't clear. – user370967 Mar 7 at 9:41
• You say that $P(X_n = x)>0$ for some n does not imply that it holds for all $n$. But in this case $P(X_{n+1} = y\mid X_n = x)$ exists and by stationarity it exists for all $n$ and is equal to this probability. In particular, $P(X_n = x)>0$ for all n in order to make the conditional probability defined. Where do I go wrong? – user370967 Mar 7 at 9:48