# Martingale property of optimal control

I am trying to solve Exercise 25.4 of Tomas Björk's Arbitrage Theory in Continuous Time.

The exercise goes as follows:

Consider the problem of minimizing

$$\mathbb{E}\left[\int_0^T F(t, X_t^u, u_t)dt + \Phi(X_T^u)\right]$$ subject to

$$dX_t = \mu(t, X_t, u_t)dt + \sigma(t, X_t, u_t)dW_t$$ and $$u(t, x) \in U$$. Here $$u$$ is some control law, $$U$$ is some constraint on the control law, $$\Phi$$ is some terminal cost function and $$F$$ is a running cost function. Define the total cost process $$C(t, u)$$ by

$$C(t, u) = \int_0^t F(s, X_s^u, u_s)ds + \mathbb{E}_{t, X_t^u}\left[\int_t^T F(t, X_t^u, u_t)dt + \Phi(X_T^u)\right].\tag{1}$$ Then, I am to show that

(a) If $$u$$ is an arbitrary control law, then $$C$$ is a submartingale.

(b) If $$u$$ is an optimal control law, then $$C$$ is a martingale.

I already got confused in part (a), as I seem to be getting that $$C$$ is a martingale even for an arbitrary $$u$$. To do this, I take conditional expectations of (1), split the first integral according to the smaller information set, use law of iterated expectations on the second integral, split the inner integral again according to the information set. Things then cancel out and I get the martingale. (If you would like me to elaborate, please ask. Seeing as it is obviously wrong, I didn't bother)

Obviously, I somehow need to use the (sub-)optimality of $$u$$, but I'm not seeing how. Any help would be greatly appreciated, thanks!

• Are you sure that the definition of $C$ is correct, and that there isn't supposed to be some sort of $\min_u$ before the conditional expectation? Feb 28, 2023 at 15:01
• Yes, at least according to the definition given in the exercise. Feb 28, 2023 at 15:28
• Just to make sure, why exactly do you expect the $\min_u$ to be there? @user6247850 Mar 2, 2023 at 9:28
• Having the $\min_u$ there would make this look more like the formulation of the dynamic programming principle I am familiar with. Without it, I agree that it looks like a martingale for arbitrary $u$. Mar 2, 2023 at 14:32

As the commenters have suggested, there seems to be an error in this exercise. Suppose instead of equation ($$1$$), $$C(t, u)$$ was defined as
$$C(t, u) = \int_0^t F(s, X_s^u, u_s)ds + V(t, X_t^u),$$ where $$V(t, X_t^u)$$ is the optimal value function. In other words, it is the last term in (1) with a $$\min_{u \in U}$$ prefix. Then, by Ito's Lemma
\begin{align} dC_t &= F(t, X_t^u, u_t)dt + \frac{\partial V}{\partial t}(t, X_t^u) + dV_t\\ &= \left\{F(t, X_t^u, u_t)dt + \frac{\partial V}{\partial t}(t, X_t^u) + \mathcal{A}^uV(t, X_t^u)\right\}dt + \sigma (t, X_t^u, u_t)\frac{\partial V}{\partial X_t^u} dW_t, \end{align} where $$\mathcal{A}^u$$ is the infinitesimal generator. The term in brackets is larger zero for an arbitrary control law and equal to zero for the optimal control by the HJB equation. The result then follows directly.