# Derivation of the Hamilton-Jacobi-Bellman equation in stochastic control

I am reading the book Continuous-time Stochastic Control and Optimization by Huyen Pham, and I am stuck on the derivation of the Hamilton-Jacobi-Bellman equation (§3.4.1 page 42).

My question is: why is $$\mathbb{E}M_h=0$$ in ($$\star$$)?

Specifically (notations are defined below), for $$a$$ fixed, after applying Itô's formula, we have $$v(t+h, X_{t+h}^{t,x})=v(t,x)+\int_{t}^{t+h}\left(\frac{\partial v}{\partial t}+\mathcal{L}^a v\right)(s,X_s^{t,x})\mathrm d s + \underbrace{\int_{t}^{t+h}\nabla_xv(s,X_s^{t,x})\sigma(X_s^{t,x},a)\mathrm dW_s}_{M_h} \tag{\star}$$ where the term $$M_h$$ is a local martingale. I tried (and failed) to show that $$M$$ is a true martingale by two approaches:

1. Show that $$\mathbb{E}\int_0^h\langle M, M\rangle_s \mathrm ds<\infty$$. However, I don't see how to bind the term in $$\nabla_xv(s,X^{t,x}_s)$$.
2. Use a localization step. The problem is that I then neeed some sort of dominated integrability condition on $$v$$ to use the dominated convergence theorem afterwards, and I don't think that such a condition holds without further assumptions on $$v$$ (e.g., quadratic growth).

Am I missing something? Is this more of a heuristic derivation (the § is titled "formal derivation")?

Some context. $$v$$ is the value function, which we assume to be smooth, $$f:(t,x,a)\rightarrow\mathbb{R}$$ is an integrable functional objective and $$X^{t,x}_s$$ is the unique strong solution to the SDE $$\mathrm d X_s = b(X_s,a)\mathrm ds+\sigma(X_s,a)\mathrm dW_s$$ started at $$x$$ at time $$t$$, where $$b$$ and $$\sigma$$ satisfy a uniform Lipschitz condition.

• I am stuck on exactly the same step. Were you able to find an explanation to it? Commented Jun 27, 2023 at 11:43