Hamiltonian (control theory)

The Hamiltonian of optimal control theory was developed by Lev Pontryagin as part of his maximum principle. It was inspired by, but is distinct from, the Hamiltonian of classical mechanics. Pontryagin proved that a necessary condition for solving the optimal control problem is that the control should be chosen so as to minimize the Hamiltonian. For details see Pontryagin's maximum principle.

Notation and Problem statement

A control $u(t)$ is to be chosen so as to minimize the objective function

$J(u)=\Psi (x(T))+\int _{0}^{T}L(x,u,t)dt$

where $x(t)$ is the system state, which evolves according to the state equations

${\dot {x}}=f(x,u,t)\qquad x(0)=x_{0}\quad t\in [0,T]$

and the control must satisfy the constraints

$a\leq u(t)\leq b\quad t\in [0,T]$

Definition of the Hamiltonian

$H(x,\lambda ,u,t)=\lambda ^{T}(t)f(x,u,t)+L(x,u,t)\,$

where $\lambda (t)$ is a vector of costate variables of the same dimension as the state variables $x(t)$.

For information on the properties of the Hamiltonian, see Pontryagin's maximum principle.

The Hamiltonian in discrete time

When the problem is formulated in discrete time, the Hamiltonian is defined as:

$H(x,\lambda ,u,t)=\lambda ^{T}(t+1)f(x,u,t)+L(x,u,t)\,$

and the costate equations are

$\lambda (t+1)^{\top }=-{\frac {\partial H}{\partial x}}+\lambda (t)^{\top }$

(Note that the discrete time Hamiltonian at time $t$ involves the costate variable at time $t+1.$ This small detail is essential so that when we differentiate with respect to $x$ we get a term involving $\lambda (t+1)$ on the right hand side of the costate equations. Using a wrong convention here can lead to incorrect results, i.e. a costate equation which is not a backwards difference equation).

The Hamiltonian of control compared to the Hamiltonian of mechanics

William Rowan Hamilton defined the Hamiltonian for describing the mechanics of a system. It is a function of three variables:

${\mathcal {H}}={\mathcal {H}}(p,q,t)=\langle p,{\dot {q}}\rangle -L(q,{\dot {q}},t)$

where $L$ is the Lagrangian, the extremizing of which determines the dynamics (not the Lagrangian defined above), $q$ is the state variable and ${\dot {q}}$ is its time derivative.

$p$ is the so-called "conjugate momentum", defined by

$p={\frac {\partial L}{\partial {\dot {q}}}}$

Hamilton then formulated his equations to describe the dynamics of the system as

${\frac {d}{dt}}p(t)=-{\frac {\partial }{\partial q}}{\mathcal {H}}$

${\frac {d}{dt}}q(t)=~~{\frac {\partial }{\partial p}}{\mathcal {H}}$

The Hamiltonian of control theory describes not the dynamics of a system but conditions for extremizing some scalar function thereof (the Lagrangian) with respect to a control variable $u$. As normally defined, it is a function of 4 variables

$H(q,u,p,t)=\langle p,{\dot {q}}\rangle -L(q,u,t)$

where $q$ is the state variable and $u$is the control variable with respect to that which we are extremizing.

The associated conditions for a maximum are

${\frac {dp}{dt}}=-{\frac {\partial H}{\partial q}}$

${\frac {dq}{dt}}=~~{\frac {\partial H}{\partial p}}$

${\frac {\partial H}{\partial u}}=0$

This definition agrees with that given by the article by Sussmann and Willems. (see p. 39, equation 14). Sussmann and Willems show how the control Hamiltonian can be used in dynamics e.g. for the brachistochrone problem, but do not mention the prior work of Carathéodory on this approach.

Example: Ramsey Model

Take a simplified version of the Ramsey–Cass–Koopmans model. We wish to maximize an agent's discounted lifetime utility achieved through consumption

$max\int _{0}^{\infty }e^{-\rho t}u(c(t))dt$

subject to the time evolution of capital per effective worker

${\dot {k}}={\frac {\partial k}{\partial t}}=f(k(t))-(n+\delta )k(t)-c(t)$

where $c(t)$ is period t consumption, $k(t)$ is period t capital per worker, $f(k(t))$ is period t production, $n$ is the population growth rate, $\delta$ is the capital depreciation rate, the agent discounts future utility at rate $\rho$, with $u'>0$ and $u''<0$.

Here, $k(t)$ is the state variable which evolves according to the above equation, and $c(t)$ is the control variable. The Hamiltonian becomes

$H(k,c,\mu ,t)=e^{-\rho t}u(c(t))+\mu (t){\dot {k}}=e^{-\rho t}u(c(t))+\mu (t)[f(k(t))-(n+\delta )k(t)-c(t)]$

The optimality conditions are

${\frac {\partial H}{\partial c}}=0\Rightarrow e^{-\rho t}u'(c)=\mu (t)$

${\frac {\partial H}{\partial k}}=-{\frac {\partial \mu }{\partial t}}=-{\dot {\mu }}\Rightarrow \mu (t)[f'(k)-(n+\delta )]=-{\dot {\mu }}$

If we let $u(c)=\log(c)$, then log-differentiating the first optimality condition with respect to $t$ yields

$-\rho -{\frac {\dot {c}}{c(t)}}={\frac {\dot {\mu }}{\mu (t)}}$

Inserting this equation into the second optimality condition yields

$\rho +{\frac {\dot {c}}{c(t)}}=f'(k)-(n+\delta )$

which is the Keynes–Ramsey rule or the Euler–Lagrange equation, which gives a condition for consumption in every period which, if followed, ensures maximum lifetime utility.

See also

• Ramsey-Cass-Koopmans model
• Keynes–Ramsey rule
• Euler–Lagrange equation

Further reading

• Wulwick, Nancy (1995). "The Hamiltonian Formalism and Optimal Growth Theory". In Rima, I. H. Measurement, Quantification, and Economic Analysis. London: Routledge. ISBN 0-415-08915-8.

External links

• Varaiya, P. (1998). "Lecture Notes on Optimization" (PDF) (2nd ed.). Archived from the original (PDF) on April 10, 2003.