In statistics,

probability theory Probability theory is the branch of mathematics concerned with probability. Although there are several different probability interpretations, probability theory treats the concept in a rigorous mathematical manner by expressing it through a set o ...

, and

information theory Information theory is the scientific study of the quantification, storage, and communication of information. The field was originally established by the works of Harry Nyquist and Ralph Hartley, in the 1920s, and Claude Shannon in the 1940s. ...

, a statistical distance quantifies the

distance Distance is a numerical or occasionally qualitative measurement of how far apart objects or points are. In physics or everyday usage, distance may refer to a physical length or an estimation based on other criteria (e.g. "two counties over"). ...

between two statistical objects, which can be two

random variable A random variable (also called random quantity, aleatory variable, or stochastic variable) is a mathematical formalization of a quantity or object which depends on random events. It is a mapping or a function from possible outcomes (e.g., the p ...

s, or two

probability distribution In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomeno ...

s or samples, or the distance can be between an individual sample point and a population or a wider sample of points. A distance between populations can be interpreted as measuring the distance between two

s and hence they are essentially measures of distances between

probability measure In mathematics, a probability measure is a real-valued function defined on a set of events in a probability space that satisfies measure properties such as ''countable additivity''. The difference between a probability measure and the more g ...

s. Where statistical distance measures relate to the differences between

s, these may have

statistical dependence Independence is a fundamental notion in probability theory, as in statistics and the theory of stochastic processes. Two events are independent, statistically independent, or stochastically independent if, informally speaking, the occurrence o ...

,Dodge, Y. (2003)—entry for distance and hence these distances are not directly related to measures of distances between probability measures. Again, a measure of distance between random variables may relate to the extent of dependence between them, rather than to their individual values. Statistical distance measures are not typically metrics, and they need not be symmetric. Some types of distance measures, which generalize ''squared'' distance, are referred to as (statistical) ''

divergence In vector calculus, divergence is a vector operator that operates on a vector field, producing a scalar field giving the quantity of the vector field's source at each point. More technically, the divergence represents the volume density of t ...

s''.

Terminology

Many terms are used to refer to various notions of distance; these are often confusingly similar, and may be used inconsistently between authors and over time, either loosely or with precise technical meaning. In addition to "distance", similar terms include deviance, deviation,

discrepancy Discrepancy may refer to: Mathematics * Discrepancy of a sequence * Discrepancy theory in structural modelling * Discrepancy of hypergraphs, an area of discrepancy theory * Discrepancy (algebraic geometry) Statistics * Discrepancy function in th ...

, discrimination, and

, as well as others such as contrast function and metric. Terms from

include cross entropy, relative entropy,

discrimination information Discrimination is the act of making unjustified distinctions between people based on the groups, classes, or other categories to which they belong or are perceived to belong. People may be discriminated on the basis of race, gender, age, relig ...

, and

information gain Information is an abstract concept that refers to that which has the power to inform. At the most fundamental level information pertains to the interpretation of that which may be sensed. Any natural process that is not completely random, ...

Distances as metrics

Metrics

A metric on a set ''X'' is a function (called the ''distance function'' or simply distance) ''d'' : ''X'' × ''X'' → R⁺ (where R⁺ is the set of non-negative

real number In mathematics, a real number is a number that can be used to measurement, measure a ''continuous'' one-dimensional quantity such as a distance, time, duration or temperature. Here, ''continuous'' means that values can have arbitrarily small var ...

s). For all ''x'', ''y'', ''z'' in ''X'', this function is required to satisfy the following conditions: # ''d''(''x'', ''y'') ≥ 0 ('' non-negativity'') # ''d''(''x'', ''y'') = 0 if and only if ''x'' = ''y'' (''

identity of indiscernibles The identity of indiscernibles is an ontological principle that states that there cannot be separate objects or entities that have all their properties in common. That is, entities ''x'' and ''y'' are identical if every predicate possessed by ' ...

''. Note that condition 1 and 2 together produce '' positive definiteness'') # ''d''(''x'', ''y'') = ''d''(''y'', ''x'') ('' symmetry'') # ''d''(''x'', ''z'') ≤ ''d''(''x'', ''y'') + ''d''(''y'', ''z'') (''

subadditivity In mathematics, subadditivity is a property of a function that states, roughly, that evaluating the function for the sum of two elements of the domain always returns something less than or equal to the sum of the function's values at each element. ...

'' / ''

triangle inequality In mathematics, the triangle inequality states that for any triangle, the sum of the lengths of any two sides must be greater than or equal to the length of the remaining side. This statement permits the inclusion of degenerate triangles, bu ...

'').

Generalized metrics

Many statistical distances are not metrics, because they lack one or more properties of proper metrics. For example, pseudometrics violate property (2), identity of indiscernibles;

quasimetric In mathematics, a metric space is a set together with a notion of ''distance'' between its elements, usually called points. The distance is measured by a function called a metric or distance function. Metric spaces are the most general setti ...

s violate property (3), symmetry; and

semimetric In mathematics, a metric space is a set together with a notion of ''distance'' between its elements, usually called points. The distance is measured by a function called a metric or distance function. Metric spaces are the most general setting ...

s violate property (4), the triangle inequality. Statistical distances that satisfy (1) and (2) are referred to as

Examples

Metrics

Total variation distance In probability theory, the total variation distance is a distance measure for probability distributions. It is an example of a statistical distance metric, and is sometimes called the statistical distance, statistical difference or variational dist ...

(sometimes just called "the" statistical distance) *

Hellinger distance In probability and statistics, the Hellinger distance (closely related to, although different from, the Bhattacharyya distance) is used to quantify the similarity between two probability distributions. It is a type of ''f''-divergence. The Hel ...

* Lévy–Prokhorov metric * Wasserstein metric: also known as the Kantorovich metric, or earth mover's distance *

Mahalanobis distance The Mahalanobis distance is a measure of the distance between a point ''P'' and a distribution ''D'', introduced by P. C. Mahalanobis in 1936. Mahalanobis's definition was prompted by the problem of identifying the similarities of skulls based ...

Divergences

Kullback–Leibler divergence In mathematical statistics, the Kullback–Leibler divergence (also called relative entropy and I-divergence), denoted D_\text(P \parallel Q), is a type of statistical distance: a measure of how one probability distribution ''P'' is different fro ...

* Rényi's divergence * Jensen–Shannon divergence * Bhattacharyya distance (despite its name it is not a distance, as it violates the triangle inequality) * f-divergence: generalizes several distances and divergences * Discriminability index, specifically the Bayes discriminability index is a positive-definite symmetric measure of the overlap of two distributions.

Notes

External links

References

*Dodge, Y. (2003) ''Oxford Dictionary of Statistical Terms'', OUP. {{ISBN, 0-19-920613-9

Terminology

Distances as metrics

Metrics

Generalized metrics

Examples

Metrics

Divergences

See also

Notes

External links

References