Convexity of Hamiltonian in optimal control Consider the classical Hamiltonian
$$H(x, p) = \sup _{u \in U} \left\{ -f(x, u) \cdot p-f^0(x, u) \right\}$$
which comes from an infinite-horizon control problem of ODE's with state equation, i.e.,
\begin{equation}\label{state_eq_finite_dim}
\left\{\begin{array}{l}
y^{\prime}(t)=f(y(t), u(t)), \quad t > 0 \\\\
y(0)=x \in \mathbb{R}^n
\end{array}\right.
\end{equation}
where $u \in \mathcal{U} := \{ u : [0,+\infty) \to U \text{ measurable} \}$ and $U \subset \mathbb{R}^n$.
The goal is to minimize the following objective
\begin{equation}
J(x, u)=\int_{0}^{+\infty} e^{-\lambda t} f^0 \left( y_x (t), u (t) \right) \,{\rm d} t
\end{equation}
Assume for simplicity that $f,f^0 \in C^1_B$ and $f$ is Lipchitz uniformly in $u$. In

*

*Martino Bardi, Italo Capuzzo Dolcetta, Optimal control and viscosity solutions of Hamilton-Jacobi-Bellman equations, Springer Science & Business Media, 1997.

it is claimed that for fixed $x$ the function $H(x,p)$ is convex in $p$. How do you see that?
 A: The Hamiltonian is convex in $p$ because it is affine (linear + constant) with respect this variable. The Hamiltonian is always affine with respect to the dual variables, regardless of the problem at hand.
Edit 1:
The supremum w.r.t. $u$ is inconsequential on the relationship w.r.t. $p$. Saying "the supremum w.r.t. $u$" is equivalent of saying "for fixed $u$".
Edit 2:
You don't need to take the supremum. You can igrone the supremum. Just take the partial derivative w.r.t. $p$. Is this derivative constant with respect to $p$? If so, then the function is affine with respect to $p$. And if a function is affine with respect to a variable, then it is convex with respect to that variable. It is a direct implication.
In the case of the Hamiltonian, the partial derivative with respect to the costates is always the right hand side of the equations of motion which are independent of the costates. Therefore the Hamiltonian is affine with respect to the costates. And therefore it is convex with respect to the costates. Always, no exception.
