My notes define a Markov chain in the following way:

A Markov chain is a stochastic process $\{X_n\}_{n=0}^\infty$ with a state space $\mathcal{S}$ that is at most countable and that satisfies the Markov-property:

For every $n \geq1$ and $(x_0, \dots, x_{n+1})\in \mathcal{S}^{n+2}$ with $\mathbb{P}(X_0 = x_0, \dots, X_n = x_n) > 0$ we have

$$\mathbb{P}(X_{n+1} = x_{n+1}\mid X_n = x_n, \dots, X_0 = x_0) = \mathbb{P}(X_{n+1} = x_{n+1}\mid X_n = x_n)$$

Moreover, the probabilities $\mathbb{P}(X_{n+1} = y \mid X_n = x)$ are stationary for all $x, y \in \mathcal{S}$ with $\mathbb{P}(X_n = x) > 0$: they do not depend on $n$ (= for all $n \geq 0$ the probabilities give the same result).


My question concerns the stationary part of the definition. Does this definition imply the following equivalence? (to make all conditional probabilities exist)

$$\exists n \geq 0: \mathbb{P}(X_n = x) > 0 \iff \forall n \geq 0: \mathbb{P}(X_n = x) >0$$

  • $\begingroup$ Does your question miss a ">0" in the end? If so, what if you consider a chain with two states (0 and 1) and where at each time the chain moves from 1 to 0 (or from 0 to 1) with probability one. $\endgroup$ – user52227 Mar 7 at 11:45
  • $\begingroup$ @user52227 I fixed it. Thanks. $\endgroup$ – user370967 Mar 7 at 12:27

My PhD is in discrete Markov chains, and I've never heard this use of terminology. What you have is a time homogeneous Markov chain. What this means is the following.

When you look at $P_n = \{ P(X_{n+1} = y \mid X_n = x) \mid x,y \in S\}$, you are looking at all the possibilities for the transition from the Markov chain's $n$-th position to its $(n+1)$-st. So first you need to look at $P_0$, then $P_1$, etc.

If $P_n$ is independent of $n$, then this means that each jump has the same distribution. Overwhelmingly such time homogeneous Markov chains are studied---although perhaps we, as a community, are wrong to not study inhomogeneous ones more...

The standard use of the term "stationarity" is very different. A distribution $\pi$ (on $S$) is called a stationary measure (my preferred term is "invariant distribution"; it's also known as an "equilibrium distribution") if the distribution of $X_n$ is $\pi$ when $X_0 \sim \pi$. So if a time-homogeneous chain has matrix $P$, then $\pi P = \pi$. It's called "equilibrium" distribution, since under certain (fairly weak) conditions one can show that, regardless of $X_0$, we have $P(X_n = j \mid X_0 = i) \to \pi(j)$ as $n \to \infty$.

Note that "$\exists n : P(X_n = x) > 0$" certain doesn't not imply that $P(X_n = x) > 0$ for all $n$. For example, take the simple Markov chain that has two vertices in the state space and always moves to vertex 1: $$ P = \begin{pmatrix} 1 & 0 \\ 1 & 0 \end{pmatrix}. $$ If you start at the second vertex, or let $X_0$ be some measure that is not a point mass at $1$, then the claim is violated: $P(X_0 = 2) > 0$ but $P(X_n = 1) = 1$ for all $n \ge 1$.

An excellent book on Markov chains is by James Norris. It really is the go-to reference, in my opinion. Grimmett also has some good stuff. (Just search for their names and "Markov chains" on your favoured search engine, aka Google.)

  • $\begingroup$ Thanks for your answer. The wikipedia page does mention "stationary markov chains", which is what I meant: en.wikipedia.org/wiki/Markov_chain#Discrete-time_Markov_chain. $\endgroup$ – user370967 Mar 6 at 20:16
  • 1
    $\begingroup$ Also, I'm sorry to inform you that this doesn't really answer my question. $\endgroup$ – user370967 Mar 6 at 20:17
  • $\begingroup$ I don't understand your question then. Maybe there's a typo in the final line? $\endgroup$ – Sam T Mar 7 at 9:38
  • $\begingroup$ Do you understand (or agree with) my definition of markov chain? If so, the question I'm asking is that if a transition probability $P(X_{n+1}=y\mid X_n = x)$ exists (i.e. $P(X_n =x)>0$), then the stationarity implies it exists for all $n$, thus does it follow that $P(X_n =x)>0$ for all $n$ in that case? ( and conversely). Sorry that the question isn't clear. $\endgroup$ – user370967 Mar 7 at 9:41
  • $\begingroup$ You say that $P(X_n = x)>0$ for some n does not imply that it holds for all $n$. But in this case $P(X_{n+1} = y\mid X_n = x)$ exists and by stationarity it exists for all $n$ and is equal to this probability. In particular, $P(X_n = x)>0$ for all n in order to make the conditional probability defined. Where do I go wrong? $\endgroup$ – user370967 Mar 7 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy