Can someone please explain the difference between dynamic optimization via the Bellman equation and dynamic optimization via Pontryagin's maximization principle?
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It only takes a minute to sign up.Sign up to join this community
The Bellman principle poses an optimization problem using a nonlinear 1st order partial differential equation - the object being optimized is a function. Pontryagin's maximum principle poses the same problem using a form of the calculus of variations - the optimized object is a curve.
This implies in technical differences and different conditions for existence of solutions, which are discussed in the literature. The optimal curve given by the maximum principle is a Cauchy characteristic of the Hamilton Jacobi Bellman partial differential equation, assuming technical conditions such that both exist.
Bellman's method was originally formulated for discrete-time systems, and extended to continuous time ones. The maximum principle is mostly studied in the continuous time framework.
From a control system theory stand point, HJB is solving for the value function - a function such that given some current state, what is the best one can do from this point onwards to minimize the cost. This is usually done backwards in time using dynamic programming. In doing so, it also solves for a control function - in a necessarily feedback form. Since the value function must be solved for all trajectories to arrive it an optimum control, the entirety of the state space is traversed. Consequentially the HJB equations are both necessary and sufficient conditions for the control to be optimal.
The maximum principle on the other hand is only a necessary condition for a control to be optimal. It finds a particular open loop control candidate; which may be one of many, each of which must be tested for optimality. For many common problems, the conditionals of the maximum principle are strong enough to easily identify optimum solutions. The maximum principle has its basic roots in the calculus of variations problems from the 1800s, like the Brachistochrone curve problem [https://en.wikipedia.org/wiki/Brachistochrone_curve], but the formalism for it, cast in the form we know today did not arrive until the 1950s.
So, in a way the maximum principle is much simpler, and tractable, but one must keep in mind many intricate caveats. The HJB equations are a lot more complex, and generally intractable without turning to concepts like generalized solution sets - viscosity solutions for example - but they are far more general, powerful and hold a lot more information (remember they put a man on the moon).
Finally as a side note, in literature, the proof of the maximum principle is quite long winded, whereas the HJB equations are quite easily proved (though requiring advanced calculus).
In these optimzation problems one needs to find a trajectory - that a curve or a sequence. Pontryagin's theory produces just this. Bellman's equation is a partial differential equation that is "woven" of Pontryagin curves - hence it looks like an overkill, unnecessary complication. I can not figure out why it is so popular. I found that Bellman's equation is used to derive in different way (almost) the same equations (they differ by the independent variable) that follow from Pontryagin's theory and then they are solved as if Bellman was never born. Bellma's principle of optimality is very charming, intuitional and handy, but it seems only a matter of taste which method one uses. What's more interesting is that the method of Lagrange's multipliers may also be used and it leads to the same conclusions as the two methods. All the three methods work well for discrete and continuous time. It seems to me that Bellman's method seems to require fewer prerequisities, that is, the teachers that use Bellmann's method do not bother to state the technical assumptions, and their students love this, intricacies of Pontyagin's theory are perhaps overwhelming.