In
statistics, M-estimators are a broad
class
Class or The Class may refer to:
Common uses not otherwise categorized
* Class (biology), a taxonomic rank
* Class (knowledge representation), a collection of individuals or objects
* Class (philosophy), an analytical concept used differently ...
of
extremum estimators for which the
objective function
In mathematical optimization and decision theory, a loss function or cost function (sometimes also called an error function) is a function that maps an event or values of one or more variables onto a real number intuitively representing some "cos ...
is a sample average. Both
non-linear least squares and
maximum likelihood estimation are special cases of M-estimators. The definition of M-estimators was motivated by
robust statistics
Robust statistics are statistics with good performance for data drawn from a wide range of probability distributions, especially for distributions that are not normal. Robust statistical methods have been developed for many common problems, such ...
, which contributed new types of M-estimators. The statistical procedure of evaluating an M-estimator on a data set is called M-estimation. 48 samples of robust M-estimators can be found in a recent review study.
More generally, an M-estimator may be defined to be a zero of an
estimating function. This estimating function is often the derivative of another statistical function. For example, a
maximum-likelihood estimate is the point where the derivative of the likelihood function with respect to the parameter is zero; thus, a maximum-likelihood estimator is a
critical point of the
score
Score or scorer may refer to:
*Test score, the result of an exam or test
Business
* Score Digital, now part of Bauer Radio
* Score Entertainment, a former American trading card design and manufacturing company
* Score Media, a former Canadian ...
function. In many applications, such M-estimators can be thought of as estimating characteristics of the population.
Historical motivation
The method of
least squares
The method of least squares is a standard approach in regression analysis to approximate the solution of overdetermined systems (sets of equations in which there are more equations than unknowns) by minimizing the sum of the squares of the re ...
is a prototypical M-estimator, since the estimator is defined as a minimum of the sum of squares of the residuals.
Another popular M-estimator is maximum-likelihood estimation. For a family of
probability density functions ''f'' parameterized by ''θ'', a
maximum likelihood
In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statis ...
estimator of ''θ'' is computed for each set of data by maximizing the
likelihood function over the parameter space . When the observations are independent and identically distributed, a ML-estimate
satisfies
:
or, equivalently,
:
Maximum-likelihood estimators have optimal properties in the limit of infinitely many observations under rather general conditions, but may be biased and not the most efficient estimators for finite samples.
Definition
In 1964,
Peter J. Huber
Peter Jost Huber (born 25 March 1934) is a Swiss statistician. He is known for his contributions to the development of heteroscedasticity-consistent standard errors.
A native of Wohlen, Aargau, Huber earned his Ph.D. at the ETH Zürich in 1962, ...
proposed generalizing maximum likelihood estimation to the minimization of
:
where ρ is a function with certain properties (see below). The solutions
:
are called M-estimators ("M" for "maximum likelihood-type" (Huber, 1981, page 43)); other types of robust estimators include
L-estimators,
R-estimators and
S-estimators. Maximum likelihood estimators (MLE) are thus a special case of M-estimators. With suitable rescaling, M-estimators are special cases of
extremum estimators (in which more general functions of the observations can be used).
The function ρ, or its derivative, ψ, can be chosen in such a way to provide the estimator desirable properties (in terms of bias and efficiency) when the data are truly from the assumed distribution, and 'not bad' behaviour when the data are generated from a model that is, in some sense, ''close'' to the assumed distribution.
Types
M-estimators are solutions, ''θ'', which minimize
:
This minimization can always be done directly. Often it is simpler to differentiate with respect to ''θ'' and solve for the root of the derivative. When this differentiation is possible, the M-estimator is said to be of ψ-type. Otherwise, the M-estimator is said to be of ρ-type.
In most practical cases, the M-estimators are of ψ-type.
ρ-type
For positive integer ''r'', let
and
be measure spaces.
is a vector of parameters. An M-estimator of ρ-type
is defined through a
measurable function
In mathematics and in particular measure theory, a measurable function is a function between the underlying sets of two measurable spaces that preserves the structure of the spaces: the preimage of any measurable set is measurable. This is in d ...
. It maps a probability distribution
on
to the value
(if it exists) that minimizes
:
:
For example, for the
maximum likelihood
In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statis ...
estimator,
, where
.
ψ-type
If
is differentiable with respect to
, the computation of
is usually much easier. An M-estimator of ψ-type ''T'' is defined through a measurable function
. It maps a probability distribution ''F'' on
to the value
(if it exists) that solves the vector equation:
:
:
For example, for the
maximum likelihood
In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statis ...
estimator,
, where
denotes the transpose of vector ''u'' and
.
Such an estimator is not necessarily an M-estimator of ρ-type, but if ρ has a continuous first derivative with respect to
, then a necessary condition for an M-estimator of ψ-type to be an M-estimator of ρ-type is
. The previous definitions can easily be extended to finite samples.
If the function ψ decreases to zero as
, the estimator is called
redescending. Such estimators have some additional desirable properties, such as complete rejection of gross outliers.
Computation
For many choices of ρ or ψ, no closed form solution exists and an iterative approach to computation is required. It is possible to use standard function optimization algorithms, such as
Newton–Raphson. However, in most cases an
iteratively re-weighted least squares
The method of iteratively reweighted least squares (IRLS) is used to solve certain optimization problems with objective functions of the form of a ''p''-norm:
:\underset \sum_^n \big, y_i - f_i (\boldsymbol\beta) \big, ^p,
by an iterative met ...
fitting algorithm can be performed; this is typically the preferred method.
For some choices of ψ, specifically, ''
redescending'' functions, the solution may not be unique. The issue is particularly relevant in multivariate and regression problems. Thus, some care is needed to ensure that good starting points are chosen.
Robust
Robustness is the property of being strong and healthy in constitution. When it is transposed into a system, it refers to the ability of tolerating perturbations that might affect the system’s functional body. In the same line ''robustness'' ca ...
starting points, such as the
median as an estimate of location and the
median absolute deviation as a univariate estimate of scale, are common.
Concentrating parameters
In computation of M-estimators, it is sometimes useful to rewrite the
objective function
In mathematical optimization and decision theory, a loss function or cost function (sometimes also called an error function) is a function that maps an event or values of one or more variables onto a real number intuitively representing some "cos ...
so that the dimension of parameters is reduced. The procedure is called “concentrating” or “profiling”. Examples in which concentrating parameters increases computation speed include
seemingly unrelated regressions In econometrics, the seemingly unrelated regressions (SUR) or seemingly unrelated regression equations (SURE) model, proposed by Arnold Zellner in (1962), is a generalization of a linear regression model that consists of several regression equatio ...
(SUR) models.
Consider the following M-estimation problem:
:
Assuming differentiability of the function ''q'', M-estimator solves the first order conditions:
:
:
Now, if we can solve the second equation for γ in terms of
and
, the second equation becomes:
:
where g is, there is some function to be found. Now, we can rewrite the original objective function solely in terms of β by inserting the function g into the place of
. As a result, there is a reduction in the number of parameters.
Whether this procedure can be done depends on particular problems at hand. However, when it is possible, concentrating parameters can facilitate computation to a great degree. For example, in estimating
SUR model of 6 equations with 5 explanatory variables in each equation by Maximum Likelihood, the number of parameters declines from 51 to 30.
Despite its appealing feature in computation, concentrating parameters is of limited use in deriving asymptotic properties of M-estimator.
The presence of W in each summand of the objective function makes it difficult to apply the
law of large numbers and the
central limit theorem
In probability theory, the central limit theorem (CLT) establishes that, in many situations, when independent random variables are summed up, their properly normalized sum tends toward a normal distribution even if the original variables themselv ...
.
Properties
Distribution
It can be shown that M-estimators are asymptotically normally distributed. As such,
Wald-type approaches to constructing confidence intervals and hypothesis tests can be used. However, since the theory is asymptotic, it will frequently be sensible to check the distribution, perhaps by examining the permutation or
bootstrap distribution.
Influence function
The influence function of an M-estimator of
-type is proportional to its defining
function.
Let ''T'' be an M-estimator of ψ-type, and ''G'' be a probability distribution for which
is defined. Its influence function IF is
:
assuming the density function
exists. A proof of this property of M-estimators can be found in Huber (1981, Section 3.2).
Applications
M-estimators can be constructed for location parameters and scale parameters in univariate and multivariate settings, as well as being used in robust regression.
Examples
Mean
Let (''X''
1, ..., ''X''
''n'') be a set of
independent, identically distributed
In probability theory and statistics, a collection of random variables is independent and identically distributed if each random variable has the same probability distribution as the others and all are mutually independent. This property is usual ...
random variables, with distribution ''F''.
If we define
:
we note that this is minimized when ''θ'' is the
mean of the ''X''s. Thus the mean is an M-estimator of ρ-type, with this ρ function.
As this ρ function is continuously differentiable in ''θ'', the mean is thus also an M-estimator of ψ-type for ψ(''x'', ''θ'') = ''θ'' − ''x''.
Median
For the median estimation of (''X''
1, ..., ''X''
''n''), instead we can define the ρ function as
:
and similarly, the ρ function is minimized when ''θ'' is the
median of the ''X''s.
While this ρ function is not differentiable in ''θ'', the ψ-type M-estimator, which is the
subgradient
In mathematics, the subderivative, subgradient, and subdifferential generalize the derivative to convex functions which are not necessarily differentiable. Subderivatives arise in convex analysis, the study of convex functions, often in connecti ...
of ρ function, can be expressed as
:
and
:
See also
*
Two-step M-estimators
*
Robust statistics
Robust statistics are statistics with good performance for data drawn from a wide range of probability distributions, especially for distributions that are not normal. Robust statistical methods have been developed for many common problems, such ...
*
Robust regression
In robust statistics, robust regression seeks to overcome some limitations of traditional regression analysis. A regression analysis models the relationship between one or more independent variables and a dependent variable. Standard types of regr ...
*
Redescending M-estimator
*
S-estimator
*
Fréchet mean
References
Further reading
*
*
*
*
*
*
*
*
*
*
*
*
*
*
External links
M-estimators— an introduction to the subject by Zhengyou Zhang
{{DEFAULTSORT:M-Estimator
M-estimators
Estimator
Robust regression
Robust statistics