machine learning Machine learning (ML) is a field of inquiry devoted to understanding and building methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. It is seen as a part of artificial intelligence. Machine ...

the random subspace method, also called attribute bagging or feature bagging, is an

ensemble learning In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Unlike a statistical ensemble in statisti ...

method that attempts to reduce the

correlation In statistics, correlation or dependence is any statistical relationship, whether causal or not, between two random variables or bivariate data. Although in the broadest sense, "correlation" may indicate any type of association, in statistics ...

between

estimator In statistics, an estimator is a rule for calculating an estimate of a given quantity based on observed data: thus the rule (the estimator), the quantity of interest (the estimand) and its result (the estimate) are distinguished. For example, the ...

s in an ensemble by training them on random samples of

features Feature may refer to: Computing * Feature (CAD), could be a hole, pocket, or notch * Feature (computer vision), could be an edge, corner or blob * Feature (software design) is an intentional distinguishing characteristic of a software item ...

instead of the entire feature set.

Motivation

In ensemble learning one tries to combine the models produced by several ''learners'' into an ''ensemble'' that performs better than the original learners. One way of combining learners is

bootstrap aggregating Bootstrap aggregating, also called bagging (from bootstrap aggregating), is a machine learning ensemble meta-algorithm designed to improve the stability and accuracy of machine learning algorithms used in statistical classification and regress ...

or ''bagging'', which shows each learner a randomly sampled subset of the training points so that the learners will produce different

models A model is an informative representation of an object, person or system. The term originally denoted the plans of a building in late 16th-century English, and derived via French and Italian ultimately from Latin ''modulus'', a measure. Models c ...

that can be sensibly averaged. In bagging, one samples training points

with replacement In statistics, quality assurance, and survey methodology, sampling is the selection of a subset (a statistical sample) of individuals from within a statistical population to estimate characteristics of the whole population. Statisticians attem ...

from the full training set. The random subspace method is similar to bagging except that the

("attributes", "predictors", "independent variables") are randomly sampled, with replacement, for each learner. Informally, this causes individual learners to not over-focus on features that appear highly predictive/descriptive in the training set, but fail to be as predictive for points outside that set. For this reason, random subspaces are an attractive choice for high-dimensional problems where the number of features is much larger than the number of training points, such as learning from fMRI data or gene expression data. The random subspace method has been used for

decision trees A decision tree is a decision support tool that uses a tree-like model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility. It is one way to display an algorithm that only contains condit ...

; when combined with "ordinary" bagging of decision trees, the resulting models are called

random forest Random forests or random decision forests is an ensemble learning method for classification, regression and other tasks that operates by constructing a multitude of decision trees at training time. For classification tasks, the output of th ...

s. It has also been applied to

linear classifier In the field of machine learning, the goal of statistical classification is to use an object's characteristics to identify which class (or group) it belongs to. A linear classifier achieves this by making a classification decision based on the val ...

support vector machine In machine learning, support vector machines (SVMs, also support vector networks) are supervised learning models with associated learning algorithms that analyze data for classification and regression analysis. Developed at AT&T Bell Laboratorie ...

s, nearest neighbours and other types of classifiers. This method is also applicable to one-class classifiers. The random subspace method has also been applied to portfolio selection problem showing its superiority to the conventional resampled portfolio essentially based on Bagging. To tackle high-dimensional sparse problems, a framework name
Random Subspace Ensemble (RaSE)
ref name=":0"> was developed. RaSE combines weak learners trained in random subspaces with a two-layer structure and iterative process. RaSE has been shown to enjoy appealing theoretical properties and practical performances.

Algorithm

An ensemble of models employing the random subspace method can be constructed using the following

algorithm In mathematics and computer science, an algorithm () is a finite sequence of rigorous instructions, typically used to solve a class of specific Computational problem, problems or to perform a computation. Algorithms are used as specificat ...

: # Let the number of training points be ''N'' and the number of features in the training data be ''D''. # Let ''L'' be the number of individual models in the ensemble. # For each individual model ''l'', choose ''n (n < N)'' to be the number of input points for l. It is common to have only one value of n for all the individual models. # For each individual model l, create a training set by choosing ''d ''features from D with replacement and train the model. Now, to apply the ensemble model to an unseen point, combine the outputs of the ''L'' individual models by majority voting or by combining the

posterior probabilities The posterior probability is a type of conditional probability that results from updating the prior probability with information summarized by the likelihood via an application of Bayes' rule. From an epistemological perspective, the posterior ...

Footnotes

References

{{DEFAULTSORT:Random Subspace Method Classification algorithms Ensemble learning