Optimal control question: find value function and optimal input If we have a continuous-time system with a scalar state variable, plant equation $$\dot{x}= u,$$ and cost function $$Q\int_o^h u^2 dt + x(h)^2,$$ then by writing the dynamic programming equation in infinitesimal form and taking a limit, I wish to show that the value function $F$ satisfies
$$ 0 = \frac{\partial{F}}{\partial{t}} + \inf_u \left[Qu^2 + \frac{\partial{F}}{\partial{x}}u\right].$$
I then want to show that $F$ and the optimal control with time $s$ to go are
$$F=\frac{Qx^2}{Q+s}$$ and $$u = - \frac{x}{Q+s}.$$
Any help with this question would be really appreciated. I am finding optimisation in continuous time very difficult.
 A: The real work here is in deriving the HJB equation. I'll split your cost up and define the variables
\begin{equation}
L(t, x, u) = Qu^2
\end{equation}
and
\begin{equation}
K(x(h)) = x(h)^2.
\end{equation}
Then your cost is just $L + K$. 
The infinitesimal form you're looking for is
\begin{equation}
F(t + \Delta t, x + \Delta x) = F(t, x) + F_t(t, x)\Delta t + F_x(t, x)\Delta x,
\end{equation}
where the equality really only holds up to higher-order terms, and the subscripts on $F$ denote partial derivatives. Rearranging gives
\begin{equation}
0 = F(t, x) - F(t + \Delta t, x + \Delta x) + F_t(t, x)\Delta t + F_x(t, x)\Delta x.
\end{equation}
Now, $F(t, x)$ gives the optimal cost to go from $t$ to $h$ and $F(t + \Delta t, x + \Delta x)$ gives the same information but from $t + \Delta t$ to $h$. Then we have that
\begin{equation}
F(t, x) - F(t + \Delta t, x + \Delta x) = \int_{t}^{h} L(\tau, x, u) d\tau - \int_{t + \Delta t}^{h}L(\tau, x, u)d\tau = \int_{t}^{t + \Delta t}L(\tau, x, u) d\tau = L(t, x, u)\Delta t,
\end{equation}
where again the equality only holds up to higher-order terms. 
Substituting this above gives us
\begin{equation}
0 = L(t, x, u)\Delta t + F_t(t, x)\Delta t + F_x(t, x)\Delta x,
\end{equation}
whereupon we apply the principal of optimality to get
\begin{equation}
0 = \inf_{u}\left[L(t, x, u)\Delta t + F_t(t, x)\Delta t + F_x(t, x)\Delta x\right].
\end{equation}
Dividing by $\Delta t$ and then taking $\Delta t \to 0$ gives
\begin{equation}
0 = \inf_{u}\left[L(t, x, u) + F_t(t, x) + F_x(t, x)f(t, x, u)\right].
\end{equation}
Plugging in $L$ and $f$, we see that
\begin{equation}
0 = \inf_{u}\left[Qu^2 + F_t(t, x) + F_x(t, x)u\right],
\end{equation}
where we can use the fact that the $F_t$ term does not depend on $u$, letting us pull it out of the infimum, giving
\begin{equation}
0 = F_t(t, x) + \inf_{u}\left[Qu^2 + F_x(t, x)u\right].
\end{equation}
Note that we haven't yet used $K$ anywhere; here $K$ provides the boundary condition for $F$ so that
\begin{equation}
F(h, x) = K(x(h)).
\end{equation}
With this, you can solve the other two parts just by plugging things in. 
