When a Markov chain converges to a steady state, what kind of convergence is it?

Question

Let $A$ be a transition matrix, the steady state distribution $x$ satisfies the distribution $Ax = x$. One can prove that under certain circumstances, $$\lim_{n\rightarrow\infty}A^n q=x$$ where $q$ is a probability distribution for the states in Markov chain. What kind of convergence is this? Is this convergence almost surely, in probability, or in distribution?

Answer 1

What is being said here is just convergence in distribution, and that fact is actually somewhat vacuous. The whole situation here is that you haven't specified an actual sequence of random variables, you've only specified the sequence of distributions given by $A^n q$. A Markov chain also introduces a corresponding sequence of random variables; in particular, given an initial distribution and $\omega \in \Omega$, we can obtain a sample path. But that's another matter entirely from the usual notion of "steady state" for Markov chains. In particular, a Markov chain will typically not converge a.s.; this would mean that the sequence $X_n$ converges to a (randomly chosen) state. Since the state space is discrete, that means the sequence is eventually constant (for a fixed $\omega$). That certainly doesn't happen for, say, $A=\begin{bmatrix} 1/2 & 1/2 \\ 1/2 & 1/2 \end{bmatrix}$. In this case soon enough there will be another transition, there is no "last" transition in the sequence.

Answer 2

The statement $$\lim_{n\to\infty}A^n q=x$$ translates in Markov Chain World into the statement: If the Markov chain has transition matrix $A$ and the initial state $X_0$ of the chain has distribution $q$ [meaning $P(X_0=i)=q_i$], then $$ \lim_{n\to\infty} P(X_n=j) = x_j\qquad\text{for every $j$.}\tag1 $$ Since each $X_n$ lives on the same countable state space, (1) implies that the sequence $X_n$ converges in distribution to a random variable having distribution $x$. This follows from the fact that for discrete random variables, $P(Y_n=y)\to P(Y=y)$ for every possible $y$ implies $Y_n\to Y$ in distribution.