2
$\begingroup$

If we have a continuous-time system with a scalar state variable, plant equation $$\dot{x}= u,$$ and cost function $$Q\int_o^h u^2 dt + x(h)^2,$$ then by writing the dynamic programming equation in infinitesimal form and taking a limit, I wish to show that the value function $F$ satisfies

$$ 0 = \frac{\partial{F}}{\partial{t}} + \inf_u \left[Qu^2 + \frac{\partial{F}}{\partial{x}}u\right].$$

I then want to show that $F$ and the optimal control with time $s$ to go are

$$F=\frac{Qx^2}{Q+s}$$ and $$u = - \frac{x}{Q+s}.$$

Any help with this question would be really appreciated. I am finding optimisation in continuous time very difficult.

$\endgroup$
2
$\begingroup$

The real work here is in deriving the HJB equation. I'll split your cost up and define the variables \begin{equation} L(t, x, u) = Qu^2 \end{equation} and \begin{equation} K(x(h)) = x(h)^2. \end{equation} Then your cost is just $L + K$.

The infinitesimal form you're looking for is \begin{equation} F(t + \Delta t, x + \Delta x) = F(t, x) + F_t(t, x)\Delta t + F_x(t, x)\Delta x, \end{equation} where the equality really only holds up to higher-order terms, and the subscripts on $F$ denote partial derivatives. Rearranging gives \begin{equation} 0 = F(t, x) - F(t + \Delta t, x + \Delta x) + F_t(t, x)\Delta t + F_x(t, x)\Delta x. \end{equation} Now, $F(t, x)$ gives the optimal cost to go from $t$ to $h$ and $F(t + \Delta t, x + \Delta x)$ gives the same information but from $t + \Delta t$ to $h$. Then we have that \begin{equation} F(t, x) - F(t + \Delta t, x + \Delta x) = \int_{t}^{h} L(\tau, x, u) d\tau - \int_{t + \Delta t}^{h}L(\tau, x, u)d\tau = \int_{t}^{t + \Delta t}L(\tau, x, u) d\tau = L(t, x, u)\Delta t, \end{equation} where again the equality only holds up to higher-order terms.

Substituting this above gives us \begin{equation} 0 = L(t, x, u)\Delta t + F_t(t, x)\Delta t + F_x(t, x)\Delta x, \end{equation} whereupon we apply the principal of optimality to get \begin{equation} 0 = \inf_{u}\left[L(t, x, u)\Delta t + F_t(t, x)\Delta t + F_x(t, x)\Delta x\right]. \end{equation} Dividing by $\Delta t$ and then taking $\Delta t \to 0$ gives \begin{equation} 0 = \inf_{u}\left[L(t, x, u) + F_t(t, x) + F_x(t, x)f(t, x, u)\right]. \end{equation} Plugging in $L$ and $f$, we see that \begin{equation} 0 = \inf_{u}\left[Qu^2 + F_t(t, x) + F_x(t, x)u\right], \end{equation} where we can use the fact that the $F_t$ term does not depend on $u$, letting us pull it out of the infimum, giving \begin{equation} 0 = F_t(t, x) + \inf_{u}\left[Qu^2 + F_x(t, x)u\right]. \end{equation} Note that we haven't yet used $K$ anywhere; here $K$ provides the boundary condition for $F$ so that \begin{equation} F(h, x) = K(x(h)). \end{equation}

With this, you can solve the other two parts just by plugging things in.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.