Dual control theory is a branch of
control theory
Control theory is a field of mathematics that deals with the control of dynamical systems in engineered processes and machines. The objective is to develop a model or algorithm governing the application of system inputs to drive the system to a ...
that deals with the control of systems whose characteristics are initially unknown.
It is called ''dual'' because in controlling such a system the controller's objectives are twofold:
* (1) Action: To control the system as well as possible based on current system knowledge
* (2) Investigation: To experiment with the system so as to learn about its behavior and control it better in the future.
These two objectives may be partly in conflict.
In the context of
reinforcement learning
Reinforcement learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. Reinforcement learning is one of three basic machine ...
, this is known as the exploration-exploitation trade-off (e.g.
Multi-armed bandit#Empirical motivation).
Dual control theory was developed by
Alexander Aronovich Fel'dbaum in 1960. He showed that in principle the
optimal
Mathematical optimization (alternatively spelled ''optimisation'') or mathematical programming is the selection of a best element, with regard to some criterion, from some set of available alternatives. It is generally divided into two subfi ...
solution can be found by
dynamic programming
Dynamic programming is both a mathematical optimization method and a computer programming method. The method was developed by Richard Bellman in the 1950s and has found applications in numerous fields, from aerospace engineering to economics.
I ...
, but this is often impractical; as a result a number of methods for designing sub-optimal dual controllers have been devised.
Example
To use an
analogy
Analogy (from Greek ''analogia'', "proportion", from ''ana-'' "upon, according to" lso "against", "anew"+ ''logos'' "ratio" lso "word, speech, reckoning" is a cognitive process of transferring information or meaning from a particular subject ( ...
: if you are driving a new car you want to get to your destination cheaply and smoothly, but you also want to see how well the car accelerates, brakes and steers so as to get a better feel for how to drive it, so you will do some test manoeuvers for this purpose. Similarly a dual controller will inject a so-called probing (or exploration) signal into the system that may detract from short-term performance but will improve control in the future.
References
*
*
* {{cite journal , first = B. , last = Wittenmark , title = Adaptive Dual Control Methods: An Overview , date = June 1995 , pages = 67–72 , publisher = Lund University , citeseerx = 10.1.1.25.7446
Control theory