I have been trying to solve this problem for a while now

Prove that if $j$ is transient state, then $\displaystyle\sum_{n=1}^\infty p_{ij}^{(n)}<\infty \ \forall i \in S$, with $S$ the state space.

I know that

$P_{ij}(s)=F_{ij}(s)P_{jj}(s)$ $\implies$

$\displaystyle\sum_{n=0}^\infty p_{ij}^ns^n=\displaystyle\sum_{n=0}^\infty f_{ij}^ns^n\displaystyle\sum_{n=1}^\infty p_{jj}^ns^n$...(1)

Since $j$ is transient, $\displaystyle\sum_{n=1}^\infty p_{jj}^{n}<\infty$, so this could be useful for the second sum in the right hand of (1). But what about all?

The result seems reasonable because if $j$ is transient, the probability of returning to $j$ from any $i$ is not certain, so looks like $\displaystyle\sum_{n=1}^\infty p_{ij}^{(n)}< \forall i \in S$ must be finite.


Let $X_n$ be the corresponding Markov chain. Define $N(j)=\sum_{n=0}^\infty 1\{X_n=j\}$ (number of visits to $j$), $\tau_j=\inf\{n\ge 0:X_n=j\}$, and $(\theta_n(\omega))(m)=\omega(m+n)$ (shift operator). Then

$$\mathbb{E}_iN(j)=\mathbb{E}_i\left[\sum_{n=0}^\infty 1\{X_n=j\}\right]=\sum_{n=0}^\infty \mathbb{E}_i[1\{X_n=j\}]=\sum_{n=0}^\infty p^n(i,j)$$

Since $N(j)=0$ on $\{\tau_j=\infty\}$ (so that $N(j)=1\{\tau_j<\infty\}N(j)$ a.s.) and

$$N(j)\circ \theta_{\tau_j}=\sum_{n=\tau_j}^\infty 1\{X_n=j\}=N(j)$$

we have (using the strong Markov property, the fact that $1\{\tau_j<\infty \}\in\mathcal{F}_{\tau_j}$, and $X_{\tau_j}=j$)

$$\mathbb{E}_iN(j)=\mathbb{E}_i[1\{\tau_j<\infty\}(N(j)\circ \theta_{\tau_j})]=\mathbb{E}_i[1\{\tau_j<\infty\}\mathbb{E}_i[N(j)\circ \theta_{\tau_j}|\mathcal{F}_{\tau_j}]]=\mathbb{E}_i[1\{\tau_j<\infty\}\mathbb{E}‌​_{X_{\tau_j}}N(j)]=P_i\{\tau_j<\infty\}\mathbb{E}_jN(j)<\infty$$

which implies that $\sum_{n=0}^\infty p^n(i,j)<\infty$.

| cite | improve this answer | |
  • $\begingroup$ Hmm... this is my first course in Stochastic Processes, so I don't understand very well the last equation. Could you elaborate or explain it in more elementary terms?. The first two are somewhat clear (at least intuitively) $\endgroup$ – Fawcett512 Jul 6 '15 at 4:46
  • $\begingroup$ @Fawcett512 Added some details $\endgroup$ – d.k.o. Jul 6 '15 at 5:16
  • $\begingroup$ For the second equation observe that we don't need to count from $0$ because $X_n$ hits $j$ for the first time at $\tau_j$. $\endgroup$ – d.k.o. Jul 6 '15 at 5:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.