In
machine learning
Machine learning (ML) is a field of inquiry devoted to understanding and building methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. It is seen as a part of artificial intelligence.
Machine ...
, particularly in the creation of
artificial neural networks, ensemble averaging is the process of creating multiple models and combining them to produce a desired output, as opposed to creating just one model. Frequently an ensemble of models performs better than any individual model, because the various errors of the models "average out."
Overview
Ensemble averaging is one of the simplest types of
committee machine A committee machine is a type of artificial neural network using a divide and conquer strategy in which the responses of multiple neural networks (experts) are combined into a single response.HAYKIN, S. Neural Networks - A Comprehensive Foundation. ...
s. Along with
boosting, it is one of the two major types of static committee machines.
[Haykin, Simon. Neural networks : a comprehensive foundation. 2nd ed. Upper Saddle River N.J.: Prentice Hall, 1999.] In contrast to standard network design in which many networks are generated but only one is kept, ensemble averaging keeps the less satisfactory networks around, but with less weight.
[Hashem, S. "Optimal linear combinations of neural networks." Neural Networks 10, no. 4 (1997): 599–614.] The theory of ensemble averaging relies on two properties of artificial neural networks:
[Naftaly, U., N. Intrator, and D. Horn. "Optimal ensemble averaging of neural networks." Network: Computation in Neural Systems 8, no. 3 (1997): 283–296.]
# In any network, the bias can be reduced at the cost of increased variance
# In a group of networks, the variance can be reduced at no cost to bias
Ensemble averaging creates a group of networks, each with low bias and high variance, then combines them to a new network with (hopefully) low bias and low variance. It is thus a resolution of the
bias-variance dilemma.
[Geman, S., E. Bienenstock, and R. Doursat. "Neural networks and the bias/variance dilemma." Neural computation 4, no. 1 (1992): 1–58.] The idea of combining experts has been traced back to
Pierre-Simon Laplace
Pierre-Simon, marquis de Laplace (; ; 23 March 1749 – 5 March 1827) was a French scholar and polymath whose work was important to the development of engineering, mathematics, statistics, physics, astronomy, and philosophy. He summarized ...
.
Method
The theory mentioned above gives an obvious strategy: create a set of experts with low bias and high variance, and then average them. Generally, what this means is to create a set of experts with varying parameters; frequently, these are the initial synaptic weights, although other factors (such as the learning rate, momentum etc.) may be varied as well. Some authors recommend against varying weight decay and early stopping.
The steps are therefore:
# Generate ''N'' experts, each with their own initial values. (Initial values are usually chosen randomly from a distribution.)
# Train each expert separately.
# Combine the experts and average their values.
Alternatively,
domain knowledge
Domain knowledge is knowledge of a specific, specialized discipline or field, in contrast to general (or domain-independent) knowledge. The term is often used in reference to a more general discipline—for example, in describing a software engin ...
may be used to generate several ''classes'' of experts. An expert from each class is trained, and then combined.
A more complex version of ensemble average views the final result not as a mere average of all the experts, but rather as a weighted sum. If each expert is
, then the overall result
can be defined as:
:
where
is a set of weights. The optimization problem of finding alpha is readily solved through neural networks, hence a "meta-network" where each "neuron" is in fact an entire neural network can be trained, and the synaptic weights of the final network is the weight applied to each expert. This is known as a ''linear combination of experts''.
It can be seen that most forms of neural networks are some subset of a linear combination: the standard neural net (where only one expert is used) is simply a linear combination with all
and one
. A raw average is where all
are equal to some constant value, namely one over the total number of experts.
A more recent ensemble averaging method is negative correlation learning, proposed by Y. Liu and X. Yao. Now this method has been widely used in
evolutionary computing
In computer science, evolutionary computation is a family of algorithms for global optimization inspired by biological evolution, and the subfield of artificial intelligence and soft computing studying these algorithms. In technical terms, the ...
.
Benefits
* The resulting committee is almost always less complex than a single network that would achieve the same level of performance
[Pearlmutter, B. A., and R. Rosenfeld. "Chaitin–Kolmogorov complexity and generalization in neural networks." In Proceedings of the 1990 conference on Advances in neural information processing systems 3, 931. Morgan Kaufmann Publishers Inc., 1990.]
* The resulting committee can be trained more easily on smaller input sets
* The resulting committee often has improved performance over any single network
* The risk of
overfitting is lessened, as there are fewer parameters (weights) which need to be set
See also
*
Ensemble learning
In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone.
Unlike a statistical ensemble in statist ...
References
Further reading
*
*
*
* {{citation , author=Hashem, S. and B. Schmeiser , title=Approximating a function and its derivatives using MSE-optimal linear combinations of trained feedforward neural networks, journal=Proceedings of the Joint Conference on Neural Networks, volume= 87, pages=617–620, year= 1993
Artificial intelligence