I am reading the book Continuous-time Stochastic Control and Optimization by Huyen Pham, and I am stuck on the derivation of the Hamilton-Jacobi-Bellman equation (§3.4.1 page 42).

My question is: why is $\mathbb{E}M_h=0$ in ($\star$)?

Specifically (notations are defined below), for $a$ fixed, after applying Itô's formula, we have $$v(t+h, X_{t+h}^{t,x})=v(t,x)+\int_{t}^{t+h}\left(\frac{\partial v}{\partial t}+\mathcal{L}^a v\right)(s,X_s^{t,x})\mathrm d s + \underbrace{\int_{t}^{t+h}\nabla_xv(s,X_s^{t,x})\sigma(X_s^{t,x},a)\mathrm dW_s}_{M_h} \tag{$\star$}$$ where the term $M_h$ is a local martingale. I tried (and failed) to show that $M$ is a true martingale by two approaches:

  1. Show that $\mathbb{E}\int_0^h\langle M, M\rangle_s \mathrm ds<\infty$. However, I don't see how to bind the term in $\nabla_xv(s,X^{t,x}_s)$.
  2. Use a localization step. The problem is that I then neeed some sort of dominated integrability condition on $v$ to use the dominated convergence theorem afterwards, and I don't think that such a condition holds without further assumptions on $v$ (e.g., quadratic growth).

Am I missing something? Is this more of a heuristic derivation (the § is titled "formal derivation")?

Some context. $v$ is the value function, which we assume to be smooth, $f:(t,x,a)\rightarrow\mathbb{R}$ is an integrable functional objective and $X^{t,x}_s$ is the unique strong solution to the SDE $$\mathrm d X_s = b(X_s,a)\mathrm ds+\sigma(X_s,a)\mathrm dW_s$$ started at $x$ at time $t$, where $b$ and $\sigma$ satisfy a uniform Lipschitz condition.

  • $\begingroup$ I am stuck on exactly the same step. Were you able to find an explanation to it? $\endgroup$ Commented Jun 27, 2023 at 11:43


You must log in to answer this question.