# Use the strong Markov property to show that the $k$th return time to a state is finite

Let

• $$(\Omega,\mathcal A,\operatorname P)$$ be a probability space
• $$(E,\mathcal E)$$ be a measureable space
• $$\kappa$$ be a Markov kernel on $$(E,\mathcal E)$$
• $$(X_n)_{n\in\mathbb N_0}$$ be an $$(E,\mathcal E)$$-valued time-homogeneous Markov chain on $$(\Omega,\mathcal A,\operatorname P)$$ with transition kernel $$\kappa$$

Let $$\sigma_B^{(0)}:=0$$ and $$\sigma_B^{(k)}:=\inf\left\{n>\sigma_B^{(k-1)}:X_n\in B\right\}$$ denote the $$k$$th return time to the state $$B\in\mathcal E$$ for $$k\in\mathbb N$$. How can we show that if $$B\in\mathcal E$$ and $$\sigma_B:=\sigma_B^{(1)}<\infty\;\;\;\text{almost surely}\tag1,$$ then $$\sigma_B^{(k)}<\infty\;\;\;\text{almost surely}\tag2$$ for all $$k\in\mathbb N$$?

Let $$k\in\mathbb N$$ with $$k\ge2$$. By $$(1)$$, we may assume that $$(2)$$ holds for $$k$$ replaced by $$k-1$$. I guess that we now need to use the strong Markov property:

Strong Markov property: If $$\tau$$ is a stopping time with respect to the filtration $$\mathcal F$$ generated by $$X$$, then$$^1$$ $$\operatorname E\left[f\left(\left(X_{\tau+k}\right)_{k\in\mathbb N_0}\right)\mid\mathcal F_\tau\right]=\left(\kappa_{\mathbb N_0}f\right)(X_\tau)\;\;\;\text{almost surely on }\{\tau<\infty\}\tag3$$ for all bounded and $$\mathcal E^{\otimes\mathbb N_0}$$-measurable $$f:E^{\mathbb N_0}\to\mathbb R$$, where $$\kappa_{\mathbb N_0}$$ is a Markov kernel$$^2$$ with source $$(E,\mathcal E)$$ and target $$\left(E^{\mathbb N_0},\mathcal E^{\otimes\mathbb N_0}\right)$$ with $$\kappa_{\mathbb N_0}(x,\;\cdot\;)\circ\left(\pi_{n_0},\ldots,\pi_{n_k}\right)^{-1}=\delta_x\otimes\bigotimes_{i=1}^k\kappa^{n_i-n_{i-1}}\tag4$$ for all $$k\in\mathbb N$$ and $$0=n_0<\cdots, where $$\pi_n$$ is the projection of $$E^{\mathbb N_0}$$ onto the $$n$$th coordinate for $$n\in\mathbb N_0$$ and $$\delta$$ is the Dirac kernel on $$(E,\mathcal E)$$.

How do we need to apply $$(3)$$? Clearly, $$\left\{\sigma_B^{(k)}<\infty\right\}=\left\{\exists n\in\mathbb N:\left(X_{\sigma_B^{(k-1)}+0},\ldots,X_{\sigma_B^{(k-1)}+n}\right)\in B\times(E\setminus B)^{n-1}\times B\right\}\tag5$$ and $$$$\begin{split}&\operatorname P\left[\left(X_{\sigma_B^{(k-1)}+0},\ldots,X_{\sigma_B^{(k-1)}+n}\right)\in B\times(E\setminus B)^{n-1}\times B\mid\mathcal F_{\sigma_B^{(k-1)}}\right]\\&\;\;\;\;\;\;\;\;\;\;\;\;=\kappa_{\mathbb N_0}\left(X_{\sigma_B^{(k-1)}},\left\{y\in E^{\mathbb N_0}:(y_0,\ldots,y_n)\in B\times(E\setminus B)^{n-1}\times B\right\}\right)\end{split}\tag6$$$$ for all $$n\in\mathbb N$$.

$$^1$$ We choose an arbitrary $$(E,\mathcal E)$$-valued random variable $$X_\infty$$ on $$(\Omega,\mathcal A,\operatorname P)$$ which is measure with respect to $$\mathcal F_\infty:=\sigma(\mathcal F_n:n\in\mathbb N_0)$$ to ensure well-definedness of $$X_\tau$$.

$$^2$$ If, for example, $$E$$ is a Polish space and $$\mathcal E=\mathcal B(E)$$, $$\kappa_{\mathbb N_0}$$ is guaranteed to exist.

Edit: Ugh, I didn't read your post carefully and instead discussed the expectations being finite. The idea for almost sure finiteness is similar, but I'll have to fix it another time. I'm leaving the post here in case it is of use.

When applying the strong Markov property, I find a few things useful to keep in mind:

1. Use the shift operator, $$\theta_t$$, defined on $$E^\mathbb{N}\rightarrow E^\mathbb{N}$$ for each $$\omega = (\omega_0, \omega_1, \dots)$$ by $$\theta_t(\omega) = (\omega_1, \omega_2, \dots).$$
2. Consider the family of probability measures $$\{P_x\}_{x\in E}$$, where $$P_x(X_0 = x) = 1$$.
3. Keep in mind that the Markov property has two parts: a part that says conditioning on $$\mathcal{F}_n$$ yields the same random variable as conditioning on $$X_n$$, and the part that says $$Y_n = X_{\sigma + n}$$ is Markov with initial distribution $$\mathrm{Law}(X_\sigma)$$. See e.g. the first 3 equations of this paper.

It's very easy to make mistakes with conditioning and even harder to spot them, especially when the result is so intuitively simple. As such, I make no claim of correctness, and hope you check my reasoning line by line.

First, I do not understand your statement of the strong Markov property (what is $$f((X_{\tau + k})_{k \in \mathbb{N}_0})$$? a sequence?), so I use the standard one: $$E_x[(Y\circ\theta_\tau)1_{\tau < \infty} | \mathcal{F}_\tau] = E_{X_\tau}[Y] 1_{\tau < \infty}$$ for any $$\mathcal{F}^\infty$$-measurable positive random variable $$Y$$. See e.g. Theorem 2.12 here. I will also take the simplified situation of $$E$$ being discrete and $$B= \{b\}$$ being a singleton, so as to isolate the use of the strong Markov property; we can generalize later.

Let $$x \in E$$ and consider the process $$X$$ started at $$x$$. Relabel $$\sigma_B^{(k)}$$ to $$\sigma^k$$ and $$\sigma_B$$ to $$\sigma$$ for simplicity. I will show that $$E_x[\sigma^k] < \infty$$ given that $$E_x[\sigma^{k-1}] < \infty$$ and $$E_b[\sigma] < \infty$$. Note that

$$\sigma^k = \sigma^{k-1} + \sigma\circ \theta_{\sigma^{k-1}} \tag{1}$$ $$P_x$$-a.s. This assertion needs to be proved, but it shows how the shift operator allows you nicely chop up assertions about stopping times. Also note that $$\sigma^k$$ is $$\mathcal{F}_{\sigma^k}$$-measurable for all $$k\geq 1$$. We have \begin{align*} E_x[\sigma^k] &= E_x[E_x[\sigma^k | \mathcal{F}_{\sigma^{k-1}}]] \\ &= E_x[E_x[\sigma^{k-1} + \sigma\circ \theta_{\sigma^{k-1}} | \mathcal{F}_{\sigma^{k-1}}]]\\ &= E_x[\sigma^{k-1}] + E_x[E_x[ \sigma\circ \theta_{\sigma^{k-1}} | \mathcal{F}_{\sigma^{k-1}}]] \\ &= E_x[\sigma^{k-1}] + E_x[E_{X_{\sigma^{k-1}}}[ \sigma]] \\ &= E_x[\sigma^{k-1}] + E_x[E_b[ \sigma]] \\ &= E_x[\sigma^{k-1}] + E_b[ \sigma] < \infty. \end{align*}

The first equality follows by the tower property, the second by the decomposition $$(1)$$, the fourth by the strong Markov property, and the fifth because $$X_{\sigma^{k-1}} = b$$, $$P_x$$-a.s. I also used that $$\sigma^{k-1} < \infty$$, $$P_x$$-a.s.

Now let's say you want to generalize to a set $$B$$, as opposed to a singleton. Well, you certainly need some assumptions on the uniformity of the return times starting from all points $$b \in B$$. For instance, consider the Markov chain on $$E = \mathbb{Z}$$ which moves deterministically to the left: $$P(X_{t+1} = x_t - 1) = 1$$, and $$B=\{-10, -9, ..., -1\}$$. Then with $$X_0 = 0$$, $$\sigma^{k}-\sigma^{k-1} = 1$$ for $$1 \leq k \leq 10$$, but after that $$\sigma^k = \infty$$.