Can someone please explain the difference between dynamic optimization via the Bellman equation and dynamic optimization via Pontryagin's maximization principle?
Thanks
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
2
The Bellman principle poses an optimization problem using a nonlinear 1st order partial differential equation - the object being optimized is a function. Pontryagin's maximum principle poses the same problem using a form of the calculus of variations - the optimized object is a curve.
This implies in technical differences and different conditions for existence of solutions, which are discussed in the literature. The optimal curve given by the maximum principle is a Cauchy characteristic of the Hamilton Jacobi Bellman partial differential equation, assuming technical conditions such that both exist.
Bellman's method was originally formulated for discrete-time systems, and extended to continuous time ones. The maximum principle is mostly studied in the continuous time framework.
-
1$\begingroup$ Thanks. Also, I looked through Pontryagin's book the other day and he clearly addresses this difference. It is interesting to note that the Bellman approach does not actually constitute a mathematical solution, but rather a "good heuristic" (in Pontryagin's words). The maximum principle on the other hand is an actual mathematical solution. I wonder why this detail is never mentioned when the Bellman equation is being taught. $\endgroup$– benOct 27, 2011 at 19:38
-
6$\begingroup$ I respectfully submit that Pontryagin's words should be seen in the light of the Cold War, and perhaps of his personal prejudices, rather than from the mathematical viewpoint. $\endgroup$– PaitMar 9, 2012 at 14:01
$\begingroup$
$\endgroup$
From a control system theory stand point, HJB is solving for the value function - a function such that given some current state, what is the best one can do from this point onwards to minimize the cost. This is usually done backwards in time using dynamic programming. In doing so, it also solves for a control function - in a necessarily feedback form. Since the value function must be solved for all trajectories to arrive it an optimum control, the entirety of the state space is traversed. Consequentially the HJB equations are both necessary and sufficient conditions for the control to be optimal.
The maximum principle on the other hand is only a necessary condition for a control to be optimal. It finds a particular open loop control candidate; which may be one of many, each of which must be tested for optimality. For many common problems, the conditionals of the maximum principle are strong enough to easily identify optimum solutions. The maximum principle has its basic roots in the calculus of variations problems from the 1800s, like the Brachistochrone curve problem [https://en.wikipedia.org/wiki/Brachistochrone_curve], but the formalism for it, cast in the form we know today did not arrive until the 1950s.
So, in a way the maximum principle is much simpler, and tractable, but one must keep in mind many intricate caveats. The HJB equations are a lot more complex, and generally intractable without turning to concepts like generalized solution sets - viscosity solutions for example - but they are far more general, powerful and hold a lot more information (remember they put a man on the moon).
Finally as a side note, in literature, the proof of the maximum principle is quite long winded, whereas the HJB equations are quite easily proved (though requiring advanced calculus).
$\begingroup$
$\endgroup$
1
In these optimzation problems one needs to find a trajectory - that a curve or a sequence. Pontryagin's theory produces just this. Bellman's equation is a partial differential equation that is "woven" of Pontryagin curves - hence it looks like an overkill, unnecessary complication. I can not figure out why it is so popular. I found that Bellman's equation is used to derive in different way (almost) the same equations (they differ by the independent variable) that follow from Pontryagin's theory and then they are solved as if Bellman was never born. Bellma's principle of optimality is very charming, intuitional and handy, but it seems only a matter of taste which method one uses. What's more interesting is that the method of Lagrange's multipliers may also be used and it leads to the same conclusions as the two methods. All the three methods work well for discrete and continuous time. It seems to me that Bellman's method seems to require fewer prerequisities, that is, the teachers that use Bellmann's method do not bother to state the technical assumptions, and their students love this, intricacies of Pontyagin's theory are perhaps overwhelming.
-
3$\begingroup$ The Bellman equation is useful precisely because it describes all of the curves. This is interesting when the boundary conditions are themselves not known a priori - perhaps they are also quantities to be optimized. This happens for instance in the study of state estimators (the Kalman filter of linear control theory). So no, it is not overkill. Bellman's method needs more technical smoothness assumptions. When they are satisfied, it is indeed mor intuitive and easier to understand. $\endgroup$– PaitDec 26, 2013 at 15:31