Similarity learning is an area of
supervised machine learning in
artificial intelligence
Artificial intelligence (AI) is intelligence—perceiving, synthesizing, and inferring information—demonstrated by machines, as opposed to intelligence displayed by animals and humans. Example tasks in which this is done include speech re ...
. It is closely related to
regression
Regression or regressions may refer to:
Science
* Marine regression, coastal advance due to falling sea level, the opposite of marine transgression
* Regression (medicine), a characteristic of diseases to express lighter symptoms or less extent ( ...
and
classification Classification is a process related to categorization, the process in which ideas and objects are recognized, differentiated and understood.
Classification is the grouping of related facts into classes.
It may also refer to:
Business, organizat ...
, but the goal is to learn a
similarity function
In statistics and related fields, a similarity measure or similarity function or similarity metric is a real-valued function that quantifies the similarity between two objects. Although no single definition of a similarity exists, usually such meas ...
that measures how
similar or related two objects are. It has applications in
ranking
A ranking is a relationship between a set of items such that, for any two items, the first is either "ranked higher than", "ranked lower than" or "ranked equal to" the second.
In mathematics, this is known as a weak order or total preorder of ...
, in
recommendation systems
A recommender system, or a recommendation system (sometimes replacing 'system' with a synonym such as platform or engine), is a subclass of information filtering system that provide suggestions for items that are most pertinent to a particular u ...
, visual identity tracking, face verification, and speaker verification.
Learning setup
There are four common setups for similarity and metric distance learning.
; ''
Regression
Regression or regressions may refer to:
Science
* Marine regression, coastal advance due to falling sea level, the opposite of marine transgression
* Regression (medicine), a characteristic of diseases to express lighter symptoms or less extent ( ...
similarity learning''
: In this setup, pairs of objects are given
together with a measure of their similarity
. The goal is to learn a function that approximates
for every new labeled triplet example
. This is typically achieved by minimizing a regularized loss
.
; ''
Classification Classification is a process related to categorization, the process in which ideas and objects are recognized, differentiated and understood.
Classification is the grouping of related facts into classes.
It may also refer to:
Business, organizat ...
similarity learning''
: Given are pairs of similar objects
and non similar objects
. An equivalent formulation is that every pair
is given together with a binary label
that determines if the two objects are similar or not. The goal is again to learn a classifier that can decide if a new pair of objects is similar or not.
; ''Ranking similarity learning''
: Given are triplets of objects
whose relative similarity obey a predefined order:
is known to be more similar to
than to
. The goal is to learn a function
such that for any new triplet of objects
, it obeys
(
contrastive learning). This setup assumes a weaker form of supervision than in regression, because instead of providing an exact
measure of similarity, one only has to provide the relative order of similarity. For this reason, ranking-based similarity learning is easier to apply in real large-scale applications.
;
Locality sensitive hashing In computer science, locality-sensitive hashing (LSH) is an algorithmic technique that hashes similar input items into the same "buckets" with high probability. (The number of buckets is much smaller than the universe of possible input items.) Sinc ...
(LSH)
:
Hashes input items so that similar items map to the same "buckets" in memory with high probability (the number of buckets being much smaller than the universe of possible input items). It is often applied in
nearest neighbor search on large-scale high-dimensional data, e.g., image databases, document collections, time-series databases, and genome databases.
A common approach for learning similarity is to model the similarity function as a
bilinear form
In mathematics, a bilinear form is a bilinear map on a vector space (the elements of which are called '' vectors'') over a field ''K'' (the elements of which are called ''scalars''). In other words, a bilinear form is a function that is linear ...
. For example, in the case of ranking similarity learning, one aims to learn a matrix W that parametrizes the similarity function
. When data is abundant, a common approach is to learn a
siamese network - A deep network model with parameter sharing.
Metric learning
Similarity learning is closely related to ''distance metric learning''. Metric learning is the task of learning a distance function over objects. A
metric or
distance function has to obey four axioms:
non-negativity,
identity of indiscernibles,
symmetry and
subadditivity (or the triangle inequality). In practice, metric learning algorithms ignore the condition of identity of indiscernibles and learn a pseudo-metric.
When the objects
are vectors in
, then any matrix
in the symmetric positive semi-definite cone
defines a distance pseudo-metric of the space of x through the form
. When
is a symmetric positive definite matrix,
is a metric. Moreover, as any symmetric positive semi-definite matrix
can be decomposed as
where
and
, the distance function
can be rewritten equivalently
. The distance
corresponds to the Euclidean distance between the transformed
feature vector
In machine learning and pattern recognition, a feature is an individual measurable property or characteristic of a phenomenon. Choosing informative, discriminating and independent features is a crucial element of effective algorithms in pattern r ...
s
and
.
Many formulations for metric learning have been proposed.
Some well-known approaches for metric learning include Learning from relative comparisons
which is based on the
Triplet loss,
Large margin nearest neighbor,
Information theoretic metric learning (ITML).
In
statistics, the
covariance
In probability theory and statistics, covariance is a measure of the joint variability of two random variables. If the greater values of one variable mainly correspond with the greater values of the other variable, and the same holds for the les ...
matrix of the data is sometimes used to define a distance metric called
Mahalanobis distance The Mahalanobis distance is a measure of the distance between a point ''P'' and a distribution ''D'', introduced by P. C. Mahalanobis in 1936. Mahalanobis's definition was prompted by the problem of identifying the similarities of skulls based ...
.
Applications
Similarity learning is used in information retrieval for
learning to rank, in face verification or face identification,
and in
recommendation systems
A recommender system, or a recommendation system (sometimes replacing 'system' with a synonym such as platform or engine), is a subclass of information filtering system that provide suggestions for items that are most pertinent to a particular u ...
. Also, many machine learning approaches rely on some metric. This includes
unsupervised learning such as
clustering, which groups together close or similar objects. It also includes supervised approaches like
K-nearest neighbor algorithm which rely on labels of nearby objects to decide on the label of a new object. Metric learning has been proposed as a preprocessing step for many of these approaches.
Scalability
Metric and similarity learning naively scale quadratically with the dimension of the input space, as can easily see when the learned metric has a bilinear form
. Scaling to higher dimensions can be achieved by enforcing a sparseness structure over the matrix model, as done with HDSL,
and with COMET.
Software
metric-learnis a
free software Python
Python may refer to:
Snakes
* Pythonidae, a family of nonvenomous snakes found in Africa, Asia, and Australia
** ''Python'' (genus), a genus of Pythonidae found in Africa and Asia
* Python (mythology), a mythical serpent
Computing
* Python (pr ...
library which offers efficient implementations of several supervised and weakly-supervised similarity and metric learning algorithms. The API of metric-learn is compatible with
scikit-learn.
OpenMetricLearningis a
Python
Python may refer to:
Snakes
* Pythonidae, a family of nonvenomous snakes found in Africa, Asia, and Australia
** ''Python'' (genus), a genus of Pythonidae found in Africa and Asia
* Python (mythology), a mythical serpent
Computing
* Python (pr ...
framework to train and validate the models producing high-quality embeddings.
See also
*
Kernel method
In machine learning, kernel machines are a class of algorithms for pattern analysis, whose best known member is the support-vector machine (SVM). The general task of pattern analysis is to find and study general types of relations (for example ...
*
Learning to rank
*
Latent semantic analysis
Further reading
For further information on this topic, see the surveys on metric and similarity learning by Bellet et al.
[ and Kulis.][
]
References
{{reflist
Machine learning
Semantic relations