# Optimal control question: find value function and optimal input

If we have a continuous-time system with a scalar state variable, plant equation $$\dot{x}= u,$$ and cost function $$Q\int_o^h u^2 dt + x(h)^2,$$ then by writing the dynamic programming equation in infinitesimal form and taking a limit, I wish to show that the value function $F$ satisfies

$$0 = \frac{\partial{F}}{\partial{t}} + \inf_u \left[Qu^2 + \frac{\partial{F}}{\partial{x}}u\right].$$

I then want to show that $F$ and the optimal control with time $s$ to go are

$$F=\frac{Qx^2}{Q+s}$$ and $$u = - \frac{x}{Q+s}.$$

Any help with this question would be really appreciated. I am finding optimisation in continuous time very difficult.

## 1 Answer

The real work here is in deriving the HJB equation. I'll split your cost up and define the variables $$L(t, x, u) = Qu^2$$ and $$K(x(h)) = x(h)^2.$$ Then your cost is just $L + K$.

The infinitesimal form you're looking for is $$F(t + \Delta t, x + \Delta x) = F(t, x) + F_t(t, x)\Delta t + F_x(t, x)\Delta x,$$ where the equality really only holds up to higher-order terms, and the subscripts on $F$ denote partial derivatives. Rearranging gives $$0 = F(t, x) - F(t + \Delta t, x + \Delta x) + F_t(t, x)\Delta t + F_x(t, x)\Delta x.$$ Now, $F(t, x)$ gives the optimal cost to go from $t$ to $h$ and $F(t + \Delta t, x + \Delta x)$ gives the same information but from $t + \Delta t$ to $h$. Then we have that $$F(t, x) - F(t + \Delta t, x + \Delta x) = \int_{t}^{h} L(\tau, x, u) d\tau - \int_{t + \Delta t}^{h}L(\tau, x, u)d\tau = \int_{t}^{t + \Delta t}L(\tau, x, u) d\tau = L(t, x, u)\Delta t,$$ where again the equality only holds up to higher-order terms.

Substituting this above gives us $$0 = L(t, x, u)\Delta t + F_t(t, x)\Delta t + F_x(t, x)\Delta x,$$ whereupon we apply the principal of optimality to get $$0 = \inf_{u}\left[L(t, x, u)\Delta t + F_t(t, x)\Delta t + F_x(t, x)\Delta x\right].$$ Dividing by $\Delta t$ and then taking $\Delta t \to 0$ gives $$0 = \inf_{u}\left[L(t, x, u) + F_t(t, x) + F_x(t, x)f(t, x, u)\right].$$ Plugging in $L$ and $f$, we see that $$0 = \inf_{u}\left[Qu^2 + F_t(t, x) + F_x(t, x)u\right],$$ where we can use the fact that the $F_t$ term does not depend on $u$, letting us pull it out of the infimum, giving $$0 = F_t(t, x) + \inf_{u}\left[Qu^2 + F_x(t, x)u\right].$$ Note that we haven't yet used $K$ anywhere; here $K$ provides the boundary condition for $F$ so that $$F(h, x) = K(x(h)).$$

With this, you can solve the other two parts just by plugging things in.