0
$\begingroup$

Suppose we have an irreducible and recurrent discrete-time Markov chain with states over the finite set $\mathcal{X}$. Let $N_t (x)$ denote the number of visits to state $x$ up to time $t$. Let $\pi(x)$ denote the stationary distribution of this Markov chain, and let $X_1 \sim \pi(x)$, i.e., the Markov chain is initially at the stationary distribution. Is the following almost surely true?

$$ \lim_{T\to \infty}\frac{1}{T} N_T(x) = \pi(x) $$

Clearly we cannot directly use strong law of large numbers (at least its standard form), since $X_1, X_2, \dots$ are not independent. Intuitively it seems obvious but I could not find a way of proving it.

$\endgroup$
2
$\begingroup$

It's true. See for example Theorem 5.3 of this reference. The idea is that you can indeed apply the strong law of large numbers by splitting $N_T(x)/T$ into excursion times: considering how long it takes to come back to state $x$ when one has just reached state $x$. These excursion times are all independent of each other and hence the strong law applies.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.