I would like to solve the following stochastic dynamic programming in the discrete-case and continuous case:

Let the state variables have the following dynamics: \begin{align*} dS_t = \mu S_t dt + \sigma S_tdW_t \\ dX_t = -\alpha_t dt \end{align*} where $S_0 = s_0$ and $X_0 = N$ and $X_T = 0$. (an inventory-like problem)

We define a continuous cost function $f$ and I am interested in minimizing the value function :

\begin{align} U^{(\alpha)}(t,S,X) = \mathbb{E}_{t,s,x}\left[\int_{t}^{T}f(u,S_u,\alpha_u)du+ g(T,S_T,X_T=0)\text{ | } S_t = s, X_t = x \right] \end{align} My cost function $f$ can writen as $f(u,S_u,\alpha_u) = g(u,S_u,\alpha_u)\alpha_u$ which comes from the fact that initially I was minimizing:

\begin{align} \mathbb{E}\left[-\int_{0}^{T}{g(t,S_t,\alpha_t)dX_t} \right] \end{align}

Using the theory of dynamic programming I can end up with the following HJB:

\begin{equation} \partial_t U + \mathcal{L}U + \inf_{\alpha}{h(\alpha)} =0 \end{equation}


\begin{equation*} h(\alpha) = -\alpha\partial_XU + f(t,S,\alpha) \end{equation*} and $\mathcal{L}$ is the linear operator w.r.t $S$. Usually to solve this we minimize $h$ and reinject the optimal $\alpha$ into the HJB to end up with a PDE. But what happens if the solution is for example 0? I happen to have $f(t,S,\alpha) = g(t,S)\alpha$ or $f(t,S,\alpha) = g(t,S)\alpha^3$. What is the theory to follow in order to solve this HJB?

Now I am interested in a discrete-time framework where updating the state variable $X_t$ happens only at discrete $t_i$ ($\partial_XU = 0$).

The value function $U(t,S,X)$ satisfies a differential equation which has no dependence on $X$: \begin{align} \partial_t{U} + \mathcal{L}U &= 0 & t \in(t_{i-1},t_i), \text{ } i = 1,2...,N \end{align} The updating of $X_t$ only occurs at the trading dates. Upon buying a quantity $\alpha_ih$ at $t_i$, the targeted quantity drops to $X-\alpha_ih$ and the liquidation cost jumps with a jump condition given by the discrete-time framework of the dynamic programming principle: \begin{equation} U(t_i^-,S,X) = \inf_{\alpha_{i} | \alpha_ih \leq N-X}\left\{U(t_i^+,S,X+\alpha_ih) + \hat{f}(t_i^+,S,\alpha_ih) \right\} \end{equation} I can solve this numerically but end up having results I don't understand. For example that my control remains the same along the state variable $S$. Does it look like normal? Is it always the case for certain form of the cost function and/or the dependance of the state variable of the control?

Thank you in advance.

  • $\begingroup$ For the discrete-time case I would recommend the book: D.P. Bertsekas and S.E. Shreve, Stochastic optimal control: the discrete time case, Academic press, 1978. However, in that case you also need a discrete-time dynamical model for your system. $\endgroup$ – Pantelis Sopasakis Dec 7 '15 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.