# Optimal control question: find value function and optimal input

If we have a continuous-time system with a scalar state variable, plant equation $$\dot{x}= u,$$ and cost function $$Q\int_o^h u^2 dt + x(h)^2,$$ then by writing the dynamic programming equation in infinitesimal form and taking a limit, I wish to show that the value function $F$ satisfies

$$0 = \frac{\partial{F}}{\partial{t}} + \inf_u \left[Qu^2 + \frac{\partial{F}}{\partial{x}}u\right].$$

I then want to show that $F$ and the optimal control with time $s$ to go are

$$F=\frac{Qx^2}{Q+s}$$ and $$u = - \frac{x}{Q+s}.$$

Any help with this question would be really appreciated. I am finding optimisation in continuous time very difficult.

The real work here is in deriving the HJB equation. I'll split your cost up and define the variables \begin{equation} L(t, x, u) = Qu^2 \end{equation} and \begin{equation} K(x(h)) = x(h)^2. \end{equation} Then your cost is just $L + K$.
The infinitesimal form you're looking for is \begin{equation} F(t + \Delta t, x + \Delta x) = F(t, x) + F_t(t, x)\Delta t + F_x(t, x)\Delta x, \end{equation} where the equality really only holds up to higher-order terms, and the subscripts on $F$ denote partial derivatives. Rearranging gives \begin{equation} 0 = F(t, x) - F(t + \Delta t, x + \Delta x) + F_t(t, x)\Delta t + F_x(t, x)\Delta x. \end{equation} Now, $F(t, x)$ gives the optimal cost to go from $t$ to $h$ and $F(t + \Delta t, x + \Delta x)$ gives the same information but from $t + \Delta t$ to $h$. Then we have that \begin{equation} F(t, x) - F(t + \Delta t, x + \Delta x) = \int_{t}^{h} L(\tau, x, u) d\tau - \int_{t + \Delta t}^{h}L(\tau, x, u)d\tau = \int_{t}^{t + \Delta t}L(\tau, x, u) d\tau = L(t, x, u)\Delta t, \end{equation} where again the equality only holds up to higher-order terms.
Substituting this above gives us \begin{equation} 0 = L(t, x, u)\Delta t + F_t(t, x)\Delta t + F_x(t, x)\Delta x, \end{equation} whereupon we apply the principal of optimality to get \begin{equation} 0 = \inf_{u}\left[L(t, x, u)\Delta t + F_t(t, x)\Delta t + F_x(t, x)\Delta x\right]. \end{equation} Dividing by $\Delta t$ and then taking $\Delta t \to 0$ gives \begin{equation} 0 = \inf_{u}\left[L(t, x, u) + F_t(t, x) + F_x(t, x)f(t, x, u)\right]. \end{equation} Plugging in $L$ and $f$, we see that \begin{equation} 0 = \inf_{u}\left[Qu^2 + F_t(t, x) + F_x(t, x)u\right], \end{equation} where we can use the fact that the $F_t$ term does not depend on $u$, letting us pull it out of the infimum, giving \begin{equation} 0 = F_t(t, x) + \inf_{u}\left[Qu^2 + F_x(t, x)u\right]. \end{equation} Note that we haven't yet used $K$ anywhere; here $K$ provides the boundary condition for $F$ so that \begin{equation} F(h, x) = K(x(h)). \end{equation}