# How $h(z)=\color{blue}{\alpha} \sum_{y} p_{z y} h(y)$ follows from Markov property?

I'm reading the paper On forward improvement iteration for stopping problems by Albrecht Irle:

We consider a discrete homogeneous Markov process $$(Z_{n})$$ with respect to the underlying filtration. The measurable state space $$(S, \mathcal{S})$$ is finite. Let $$g: S \rightarrow \mathbb{R}$$ be measurable and $$\alpha \in (0,1]$$. We look at the optimal stopping problem for $$X_{n}=\alpha^ng\left(Z_{n}\right)$$

Let

• $$P_{z}, E_{z}$$ denote $$P(\cdot \mid Z_{0}=z), E(\cdot \mid Z_{0}=z)$$ respectively.
• $$E_{z} X_{\tau}$$ exist for all stopping rules $$\tau$$ and all $$z \in S$$.
• $$p_{z y}=P(Z_{1}=y \mid Z_{0}=z)$$ for all $$y, z \in S$$.
• $$\tau_{n}(B)=\inf \left\{j \geq n \mid Z_{j} \in B\right\}$$ for a measurable $$B \in \mathcal{S}$$.
• $$h_{i}(B)(z)=E_{z} \alpha^{\tau_{i}(B)} g\left(Z_{\tau_{i}(B)}\right)$$ for $$z \in S$$ and $$i \in \{0,1\}$$.

Then the author present a proposition and its proof:

I'm trying to understand how to get $$\forall z \in S \setminus B:h(z)=\color{blue}{\alpha} \sum_{y} p_{z y} h(y)$$ We have\begin{aligned} h_{0}(B)(z) &= E_z \left [ \alpha^{\tau_{0}(B)} g\left(Z_{\tau_{0}(B)}\right) \right ]\\ &= \sum _{k=0}^\infty \alpha^{k} g\left(Z_{k}\right) P_z\left [ \tau_{0}(B) = k \right ]\\ &= \sum _{k=0}^\infty \alpha^{k} g\left(Z_{k}\right) \sum_y P_z\left [ \tau_{0}(B) = k,X_1=y \right ] \\ &= \sum _{k=0}^\infty \alpha^{k} g\left(Z_{k}\right) \sum_y P_z\left [ \tau_{0}(B) = k \mid X_1=y \right ] P_z [X_1=y]\\ &= \sum _{k=0}^\infty \alpha^{k} g\left(Z_{k}\right) \sum_y P_y\left [ \tau_{0}(B) = k \right ] p_{zy}\\ &= \sum_y \left [ \sum _{k=0}^\infty \alpha^{k} g\left(Z_{k}\right) P_y\left [ \tau_{0}(B) = k \right ] \right ] p_{zy}\\ &= \sum_y E_y \left [ \alpha^{\tau_{0}(B)} g\left(Z_{\tau_{0}(B)}\right) \right ] p_{zy}\\ &= \sum_y h_{0}(B)(y) p_{zy} \end{aligned}

In my attempt, I could not see how the constant $$\color{blue}{\alpha}$$ appears. Could you please elaborate on this point?

$$\def\peq{\mathrel{\phantom{=}}{}}\def\tb{{τ_0(B)}}$$Since $$z \in S \setminus B$$, then\begin{align*} &\peq E_z\left( α^\tb g(Z_\tb) \right) = \sum_{k = \color{red}{1}}^∞ E_z\left( α^k g(Z_k) I_{\{\tb = k\}} \right)\\ &= \sum_{k = 1}^∞ \sum_{\color{red}{x \in B}} E_z(α^k g(x) I_{\{Z_k = x, \tb = k\}}) = \sum_{k = 1}^∞ \sum_{x \in B} α^k g(x) P_z(Z_k = x, \tb = k)\\ &= \sum_{k = 1}^∞ \sum_{x \in B} \sum_{y \in S} α^k g(x) P_z(Z_k = x, \tb = k \mid \color{red}{Z_1} = y) P_z(Z_1 = y).\tag{1} \end{align*} Note that $$P_z(Z_1 = y) = p_{zy}$$. For $$y \in B$$,$$P_z(Z_k = x, \tb = k \mid Z_1 = y) = \begin{cases} δ_{xy}; & k = 1\\ 0; & k \geqslant 2 \end{cases},$$ and for $$y \in S \setminus B$$,$$P_z(Z_k = x, \tb = k \mid Z_1 = y) = \begin{cases} 0; & k = 1\\ P_y(Z_{k - 1} = x, \tb = k - 1); & k \geqslant 2\\ \end{cases},$$ thus$$\begin{gather*} \sum_{x \in B} \sum_{y \in S} α g(x) P_z(Z_1 = x, \tb = 1 \mid Z_1 = y) p_{zy}\\ = α \sum_{y \in B} p_{zy} g(y) = α \sum_{y \in B} p_{zy} E_y\left( α^\tb g(Z_\tb) \right), \end{gather*}$$\begin{align*} &\peq \sum_{k = 2}^∞ \sum_{x \in B} \sum_{y \in S} α^k g(x) P_z(Z_k = x, \tb = k \mid Z_1 = y) p_{zy}\\ &= \sum_{k = 2}^∞ \sum_{x \in B} \sum_{y \in S \setminus B} α^k g(x) P_y(Z_{k - 1} = x, \tb = k - 1) p_{zy}\\ &= α \sum_{y \in S \setminus B} p_{zy} \sum_{k = 2}^∞ \sum_{x \in B} α^{k - 1} g(x) P_y(Z_{k - 1} = x, \tb = k - 1)\\ &= α \sum_{y \in S \setminus B} p_{zy} \sum_{k = 1}^∞ \sum_{x \in B} α^k g(x) P_y(Z_k = x, \tb = k)\\ &= α \sum_{y \in S \setminus B} p_{zy} \sum_{k = 1}^∞ \sum_{x \in B} E_y(α^k g(x) I_{\{Z_k = x, \tb = k\}})\\ &= α \sum_{y \in S \setminus B} p_{zy} \sum_{k = 1}^∞ E_y(α^k g(Z_k) I_{\{\tb = k\}}) = α \sum_{y \in S \setminus B} p_{zy} E_y\left( α^\tb g(Z_\tb) \right) \end{align*} and\begin{align*} (1) &= \sum_{x \in B} \sum_{y \in S} α g(x) P_z(Z_1 = x, \tb = 1 \mid Z_1 = y) p_{zy}\\ &\peq + \sum_{k = 2}^∞ \sum_{x \in B} \sum_{y \in S} α^k g(x) P_z(Z_k = x, \tb = k \mid Z_1 = y) p_{zy}\\ &= α \sum_{y \in B} p_{zy} E_y\left( α^\tb g(Z_\tb) \right) + α \sum_{y \in S \setminus B} p_{zy} E_y\left( α^\tb g(Z_\tb) \right)\\ &= α \sum_{y \in S} p_{zy} E_y\left( α^\tb g(Z_\tb) \right). \end{align*}
• @LAD The mistake mainly occurs at$$E_z\left(α^{τ_0(B)}g(Z_{τ_0(B)})\right)=\sum _{k=0}^∞α^kg(Z_k)P_z(τ_0(B)=k),$$where the RHS has $g(Z_k)$, i.e. a random variable, outside the expection. – Saad May 14 at 9:59