4
$\begingroup$

I'm reading the paper On forward improvement iteration for stopping problems by Albrecht Irle:

We consider a discrete homogeneous Markov process $(Z_{n})$ with respect to the underlying filtration. The measurable state space $(S, \mathcal{S})$ is finite. Let $g: S \rightarrow \mathbb{R}$ be measurable and $\alpha \in (0,1]$. We look at the optimal stopping problem for $$X_{n}=\alpha^ng\left(Z_{n}\right)$$

Let

  • $P_{z}, E_{z}$ denote $P(\cdot \mid Z_{0}=z), E(\cdot \mid Z_{0}=z)$ respectively.
  • $E_{z} X_{\tau}$ exist for all stopping rules $\tau$ and all $z \in S$.
  • $p_{z y}=P(Z_{1}=y \mid Z_{0}=z)$ for all $y, z \in S$.
  • $\tau_{n}(B)=\inf \left\{j \geq n \mid Z_{j} \in B\right\}$ for a measurable $B \in \mathcal{S}$.
  • $h_{i}(B)(z)=E_{z} \alpha^{\tau_{i}(B)} g\left(Z_{\tau_{i}(B)}\right)$ for $z \in S$ and $i \in \{0,1\}$.

Then the author present a proposition and its proof:


I'm trying to understand how to get $$\forall z \in S \setminus B:h(z)=\color{blue}{\alpha} \sum_{y} p_{z y} h(y)$$ We have\begin{aligned} h_{0}(B)(z) &= E_z \left [ \alpha^{\tau_{0}(B)} g\left(Z_{\tau_{0}(B)}\right) \right ]\\ &= \sum _{k=0}^\infty \alpha^{k} g\left(Z_{k}\right) P_z\left [ \tau_{0}(B) = k \right ]\\ &= \sum _{k=0}^\infty \alpha^{k} g\left(Z_{k}\right) \sum_y P_z\left [ \tau_{0}(B) = k,X_1=y \right ] \\ &= \sum _{k=0}^\infty \alpha^{k} g\left(Z_{k}\right) \sum_y P_z\left [ \tau_{0}(B) = k \mid X_1=y \right ] P_z [X_1=y]\\ &= \sum _{k=0}^\infty \alpha^{k} g\left(Z_{k}\right) \sum_y P_y\left [ \tau_{0}(B) = k \right ] p_{zy}\\ &= \sum_y \left [ \sum _{k=0}^\infty \alpha^{k} g\left(Z_{k}\right) P_y\left [ \tau_{0}(B) = k \right ] \right ] p_{zy}\\ &= \sum_y E_y \left [ \alpha^{\tau_{0}(B)} g\left(Z_{\tau_{0}(B)}\right) \right ] p_{zy}\\ &= \sum_y h_{0}(B)(y) p_{zy} \end{aligned}

In my attempt, I could not see how the constant $\color{blue}{\alpha}$ appears. Could you please elaborate on this point?

$\endgroup$
1
+500
$\begingroup$

$\def\peq{\mathrel{\phantom{=}}{}}\def\tb{{τ_0(B)}}$Since $z \in S \setminus B$, then\begin{align*} &\peq E_z\left( α^\tb g(Z_\tb) \right) = \sum_{k = \color{red}{1}}^∞ E_z\left( α^k g(Z_k) I_{\{\tb = k\}} \right)\\ &= \sum_{k = 1}^∞ \sum_{\color{red}{x \in B}} E_z(α^k g(x) I_{\{Z_k = x, \tb = k\}}) = \sum_{k = 1}^∞ \sum_{x \in B} α^k g(x) P_z(Z_k = x, \tb = k)\\ &= \sum_{k = 1}^∞ \sum_{x \in B} \sum_{y \in S} α^k g(x) P_z(Z_k = x, \tb = k \mid \color{red}{Z_1} = y) P_z(Z_1 = y).\tag{1} \end{align*} Note that $P_z(Z_1 = y) = p_{zy}$. For $y \in B$,$$ P_z(Z_k = x, \tb = k \mid Z_1 = y) = \begin{cases} δ_{xy}; & k = 1\\ 0; & k \geqslant 2 \end{cases}, $$ and for $y \in S \setminus B$,$$ P_z(Z_k = x, \tb = k \mid Z_1 = y) = \begin{cases} 0; & k = 1\\ P_y(Z_{k - 1} = x, \tb = k - 1); & k \geqslant 2\\ \end{cases}, $$ thus\begin{gather*} \sum_{x \in B} \sum_{y \in S} α g(x) P_z(Z_1 = x, \tb = 1 \mid Z_1 = y) p_{zy}\\ = α \sum_{y \in B} p_{zy} g(y) = α \sum_{y \in B} p_{zy} E_y\left( α^\tb g(Z_\tb) \right), \end{gather*}\begin{align*} &\peq \sum_{k = 2}^∞ \sum_{x \in B} \sum_{y \in S} α^k g(x) P_z(Z_k = x, \tb = k \mid Z_1 = y) p_{zy}\\ &= \sum_{k = 2}^∞ \sum_{x \in B} \sum_{y \in S \setminus B} α^k g(x) P_y(Z_{k - 1} = x, \tb = k - 1) p_{zy}\\ &= α \sum_{y \in S \setminus B} p_{zy} \sum_{k = 2}^∞ \sum_{x \in B} α^{k - 1} g(x) P_y(Z_{k - 1} = x, \tb = k - 1)\\ &= α \sum_{y \in S \setminus B} p_{zy} \sum_{k = 1}^∞ \sum_{x \in B} α^k g(x) P_y(Z_k = x, \tb = k)\\ &= α \sum_{y \in S \setminus B} p_{zy} \sum_{k = 1}^∞ \sum_{x \in B} E_y(α^k g(x) I_{\{Z_k = x, \tb = k\}})\\ &= α \sum_{y \in S \setminus B} p_{zy} \sum_{k = 1}^∞ E_y(α^k g(Z_k) I_{\{\tb = k\}}) = α \sum_{y \in S \setminus B} p_{zy} E_y\left( α^\tb g(Z_\tb) \right) \end{align*} and\begin{align*} (1) &= \sum_{x \in B} \sum_{y \in S} α g(x) P_z(Z_1 = x, \tb = 1 \mid Z_1 = y) p_{zy}\\ &\peq + \sum_{k = 2}^∞ \sum_{x \in B} \sum_{y \in S} α^k g(x) P_z(Z_k = x, \tb = k \mid Z_1 = y) p_{zy}\\ &= α \sum_{y \in B} p_{zy} E_y\left( α^\tb g(Z_\tb) \right) + α \sum_{y \in S \setminus B} p_{zy} E_y\left( α^\tb g(Z_\tb) \right)\\ &= α \sum_{y \in S} p_{zy} E_y\left( α^\tb g(Z_\tb) \right). \end{align*}

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I'm going to read your answer carefully tonight :)) $\endgroup$ – LAD May 13 at 16:16
  • $\begingroup$ I'm reading your answer. Could you please elaborate on which point that I got incorrect? $\endgroup$ – LAD May 13 at 23:45
  • $\begingroup$ @LAD The mistake mainly occurs at$$E_z\left(α^{τ_0(B)}g(Z_{τ_0(B)})\right)=\sum _{k=0}^∞α^kg(Z_k)P_z(τ_0(B)=k),$$where the RHS has $g(Z_k)$, i.e. a random variable, outside the expection. $\endgroup$ – Saad May 14 at 9:59
  • $\begingroup$ Thank you so much for your dedicated support. It seems that I've figured out another way to prove this proposition here. If you don't mind, please have a look at it. $\endgroup$ – LAD May 15 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.