In
optimal control theory
Mathematical optimization (alternatively spelled ''optimisation'') or mathematical programming is the selection of a best element, with regard to some criterion, from some set of available alternatives. It is generally divided into two subfi ...
, the Hamilton-Jacobi-Bellman (HJB) equation gives a
necessary and sufficient condition
In logic and mathematics, necessity and sufficiency are terms used to describe a conditional or implicational relationship between two statements. For example, in the conditional statement: "If then ", is necessary for , because the truth of ...
for
optimality
Optimality may refer to:
* Mathematical optimization
* Optimality Theory in linguistics
* optimality model In biology, optimality models are a tool used to evaluate the costs and benefits of different organismal features, traits, and characterist ...
of a
control
Control may refer to:
Basic meanings Economics and business
* Control (management), an element of management
* Control, an element of management accounting
* Comptroller (or controller), a senior financial officer in an organization
* Controllin ...
with respect to a
loss function. It is, in general, a nonlinear
partial differential equation in the
value function The value function of an optimization problem gives the value attained by the objective function at a solution, while only depending on the parameters of the problem. In a controlled dynamical system, the value function represents the optimal payof ...
, which means its solution the value function itself. Once this solution is known, it can be used to obtain the optimal control by taking the maximizer (or minimizer) of the
Hamiltonian
Hamiltonian may refer to:
* Hamiltonian mechanics, a function that represents the total energy of a system
* Hamiltonian (quantum mechanics), an operator corresponding to the total energy of that system
** Dyall Hamiltonian, a modified Hamiltonian ...
involved in the HJB equation.
The equation is a result of the theory of
dynamic programming
Dynamic programming is both a mathematical optimization method and a computer programming method. The method was developed by Richard Bellman in the 1950s and has found applications in numerous fields, from aerospace engineering to economics. ...
which was pioneered in the 1950s by
Richard Bellman and coworkers. The connection to the
Hamilton–Jacobi equation
In physics, the Hamilton–Jacobi equation, named after William Rowan Hamilton and Carl Gustav Jacob Jacobi, is an alternative formulation of classical mechanics, equivalent to other formulations such as Newton's laws of motion, Lagrangian mecha ...
from
classical physics was first drawn by
Rudolf Kálmán. In
discrete-time
In mathematical dynamics, discrete time and continuous time are two alternative frameworks within which variables that evolve over time are modeled.
Discrete time
Discrete time views values of variables as occurring at distinct, separate "po ...
problems, the corresponding
difference equation is usually referred to as the
Bellman equation
A Bellman equation, named after Richard E. Bellman, is a necessary condition for optimality associated with the mathematical optimization method known as dynamic programming. It writes the "value" of a decision problem at a certain point in time ...
.
While classical
variational problems, such as the
brachistochrone problem
In physics and mathematics, a brachistochrone curve (), or curve of fastest descent, is the one lying on the plane between a point ''A'' and a lower point ''B'', where ''B'' is not directly below ''A'', on which a bead slides frictionlessly under ...
, can be solved using the Hamilton–Jacobi–Bellman equation, the method can be applied to a broader spectrum of problems. Further it can be generalized to
stochastic systems, in which case the HJB equation is a second-order
elliptic partial differential equation. A major drawback, however, is that the HJB equation admits classical solutions only for a
sufficiently smooth value function, which is not guaranteed in most situations. Instead, the notion of a
viscosity solution In mathematics, the viscosity solution concept was introduced in the early 1980s by Pierre-Louis Lions and Michael G. Crandall as a generalization of the classical concept of what is meant by a 'solution' to a partial differential equation (PDE) ...
is required, in which conventional derivatives are replaced by (set-valued)
subderivative
In mathematics, the subderivative, subgradient, and subdifferential generalize the derivative to convex functions which are not necessarily differentiable. Subderivatives arise in convex analysis, the study of convex functions, often in connectio ...
s.
Optimal-control-problems
Consider the following problem in deterministic optimal control over the time period