# Properties of a transient state in a Markov Chain

I have been trying to solve this problem for a while now

Prove that if $j$ is transient state, then $\displaystyle\sum_{n=1}^\infty p_{ij}^{(n)}<\infty \ \forall i \in S$, with $S$ the state space.

I know that

$P_{ij}(s)=F_{ij}(s)P_{jj}(s)$ $\implies$

$\displaystyle\sum_{n=0}^\infty p_{ij}^ns^n=\displaystyle\sum_{n=0}^\infty f_{ij}^ns^n\displaystyle\sum_{n=1}^\infty p_{jj}^ns^n$...(1)

Since $j$ is transient, $\displaystyle\sum_{n=1}^\infty p_{jj}^{n}<\infty$, so this could be useful for the second sum in the right hand of (1). But what about all?

The result seems reasonable because if $j$ is transient, the probability of returning to $j$ from any $i$ is not certain, so looks like $\displaystyle\sum_{n=1}^\infty p_{ij}^{(n)}< \forall i \in S$ must be finite.

Let $X_n$ be the corresponding Markov chain. Define $N(j)=\sum_{n=0}^\infty 1\{X_n=j\}$ (number of visits to $j$), $\tau_j=\inf\{n\ge 0:X_n=j\}$, and $(\theta_n(\omega))(m)=\omega(m+n)$ (shift operator). Then

$$\mathbb{E}_iN(j)=\mathbb{E}_i\left[\sum_{n=0}^\infty 1\{X_n=j\}\right]=\sum_{n=0}^\infty \mathbb{E}_i[1\{X_n=j\}]=\sum_{n=0}^\infty p^n(i,j)$$

Since $N(j)=0$ on $\{\tau_j=\infty\}$ (so that $N(j)=1\{\tau_j<\infty\}N(j)$ a.s.) and

$$N(j)\circ \theta_{\tau_j}=\sum_{n=\tau_j}^\infty 1\{X_n=j\}=N(j)$$

we have (using the strong Markov property, the fact that $1\{\tau_j<\infty \}\in\mathcal{F}_{\tau_j}$, and $X_{\tau_j}=j$)

$$\mathbb{E}_iN(j)=\mathbb{E}_i[1\{\tau_j<\infty\}(N(j)\circ \theta_{\tau_j})]=\mathbb{E}_i[1\{\tau_j<\infty\}\mathbb{E}_i[N(j)\circ \theta_{\tau_j}|\mathcal{F}_{\tau_j}]]=\mathbb{E}_i[1\{\tau_j<\infty\}\mathbb{E}‌​_{X_{\tau_j}}N(j)]=P_i\{\tau_j<\infty\}\mathbb{E}_jN(j)<\infty$$

which implies that $\sum_{n=0}^\infty p^n(i,j)<\infty$.

• Hmm... this is my first course in Stochastic Processes, so I don't understand very well the last equation. Could you elaborate or explain it in more elementary terms?. The first two are somewhat clear (at least intuitively) – Fawcett512 Jul 6 '15 at 4:46
• @Fawcett512 Added some details – d.k.o. Jul 6 '15 at 5:16
• For the second equation observe that we don't need to count from $0$ because $X_n$ hits $j$ for the first time at $\tau_j$. – d.k.o. Jul 6 '15 at 5:20