Meta-optimization from
numerical optimization
Mathematical optimization (alternatively spelled ''optimisation'') or mathematical programming is the selection of a best element, with regard to some criteria, from some set of available alternatives. It is generally divided into two subfiel ...
is the use of one optimization method to tune another optimization method. Meta-optimization is reported to have been used as early as in the late 1970s by Mercer and Sampson
for finding optimal parameter settings of a
genetic algorithm
In computer science and operations research, a genetic algorithm (GA) is a metaheuristic inspired by the process of natural selection that belongs to the larger class of evolutionary algorithms (EA). Genetic algorithms are commonly used to g ...
.
Meta-optimization and related concepts are also known in the literature as meta-evolution, super-optimization, automated parameter calibration,
hyper-heuristics A hyper-heuristic is a heuristic search method that seeks to automate, often by the incorporation of machine learning techniques, the process of selecting, combining, generating or adapting several simpler heuristics (or components of such heuristic ...
, etc.
Motivation
Optimization methods such as
genetic algorithm
In computer science and operations research, a genetic algorithm (GA) is a metaheuristic inspired by the process of natural selection that belongs to the larger class of evolutionary algorithms (EA). Genetic algorithms are commonly used to g ...
and
differential evolution
Differential evolution (DE) is an evolutionary algorithm to optimize a problem by iteratively trying to improve a candidate solution with regard to a given measure of quality. Such methods are commonly known as metaheuristics as they make few ...
have several parameters that govern their behaviour and efficiency in optimizing a given problem and these parameters must be chosen by the practitioner to achieve satisfactory results. Selecting the behavioural parameters by hand is a laborious task that is susceptible to human misconceptions of what makes the optimizer perform well.
The behavioural parameters of an optimizer can be varied and the optimization performance plotted as a landscape. This is computationally feasible for optimizers with few behavioural parameters and optimization problems that are fast to compute, but when the number of behavioural parameters increases the time usage for computing such a performance landscape increases exponentially. This is the
curse of dimensionality
The curse of dimensionality refers to various phenomena that arise when analyzing and organizing data in high-dimensional spaces that do not occur in low-dimensional settings such as the three-dimensional physical space of everyday experience. T ...
for the search-space consisting of an optimizer's behavioural parameters. An efficient method is therefore needed to search the space of behavioural parameters.
Methods
A simple way of finding good behavioural parameters for an optimizer is to employ another overlaying optimizer, called the
meta
Meta most commonly refers to:
* Meta (prefix), a common affix and word in English ( in Greek)
* Meta Platforms, an American multinational technology conglomerate (formerly ''Facebook, Inc.'')
Meta or META may also refer to:
Businesses
* Meta (ac ...
-optimizer. There are different ways of doing this depending on whether the behavioural parameters to be tuned are
real-valued
In mathematics, value may refer to several, strongly related notions.
In general, a mathematical value may be any definite mathematical object. In elementary mathematics, this is most often a number – for example, a real number such as or an ...
or
discrete-valued, and depending on what performance measure is being used, etc.
Meta-optimizing the parameters of a
genetic algorithm
In computer science and operations research, a genetic algorithm (GA) is a metaheuristic inspired by the process of natural selection that belongs to the larger class of evolutionary algorithms (EA). Genetic algorithms are commonly used to g ...
was done by Grefenstette
[ and Keane,][ amongst others, and experiments with meta-optimizing both the parameters and the genetic operators were reported by Bäck.][ Meta-optimization of the COMPLEX-RF algorithm was done by Krus and Andersson,][ and,][ where performance index of optimization based on information theory was introduced and further developed. Meta-optimization of ]particle swarm optimization
In computational science, particle swarm optimization (PSO) is a computational method that Mathematical optimization, optimizes a problem by iterative method, iteratively trying to improve a candidate solution with regard to a given measure of qu ...
was done by Meissner et al.,[ Pedersen and Chipperfield,][ and Mason et al.][ Pedersen and Chipperfield applied meta-optimization to ]differential evolution
Differential evolution (DE) is an evolutionary algorithm to optimize a problem by iteratively trying to improve a candidate solution with regard to a given measure of quality. Such methods are commonly known as metaheuristics as they make few ...
.[ Birattari et al.][ meta-optimized ]ant colony optimization
In computer science and operations research, the ant colony optimization algorithm (ACO) is a probabilistic technique for solving computational problems that can be reduced to finding good paths through graphs. Artificial ants represent multi ...
. Statistical models
A statistical model is a mathematical model that embodies a set of statistical assumptions concerning the generation of sample data (and similar data from a larger population). A statistical model represents, often in considerably idealized form ...
have also been used to reveal more about the relationship between choices of behavioural parameters and optimization performance, see for example Francois and Lavergne,[ and Nannen and Eiben.][ A comparison of various meta-optimization techniques was done by Smit and Eiben.][
]
See also
* Automated machine learning
Automated machine learning (AutoML) is the process of automating the tasks of applying machine learning to real-world problems. It is the combination of automation and ML.
AutoML potentially includes every stage from beginning with a raw datas ...
(AutoML)
* Hyper-heuristics A hyper-heuristic is a heuristic search method that seeks to automate, often by the incorporation of machine learning techniques, the process of selecting, combining, generating or adapting several simpler heuristics (or components of such heuristic ...
References
{{Optimization algorithms
Evolutionary computation
Heuristics