In
statistics, an estimator is a rule for calculating an estimate of a given quantity based on
observed data: thus the rule (the estimator), the quantity of interest (the
estimand An estimand is a quantity that is to be estimated in a statistical analysis. The term is used to more clearly distinguish the target of inference from the method used to obtain an approximation of this target (i.e., the estimator) and the specific v ...
) and its result (the estimate) are distinguished. For example, the
sample mean
The sample mean (or "empirical mean") and the sample covariance are statistics computed from a sample of data on one or more random variables.
The sample mean is the average value (or mean value) of a sample of numbers taken from a larger popu ...
is a commonly used estimator of the
population mean
In statistics, a population is a set of similar items or events which is of interest for some question or experiment. A statistical population can be a group of existing objects (e.g. the set of all stars within the Milky Way galaxy) or a hypothe ...
.
There are
point
Point or points may refer to:
Places
* Point, Lewis, a peninsula in the Outer Hebrides, Scotland
* Point, Texas, a city in Rains County, Texas, United States
* Point, the NE tip and a ferry terminal of Lismore, Inner Hebrides, Scotland
* Point ...
and
interval estimators. The
point estimators yield single-valued results. This is in contrast to an
interval estimator, where the result would be a range of plausible values. "Single value" does not necessarily mean "single number", but includes vector valued or function valued estimators.
''
Estimation theory'' is concerned with the properties of estimators; that is, with defining properties that can be used to compare different estimators (different rules for creating estimates) for the same quantity, based on the same data. Such properties can be used to determine the best rules to use under given circumstances. However, in
robust statistics
Robust statistics are statistics with good performance for data drawn from a wide range of probability distributions, especially for distributions that are not normal. Robust statistical methods have been developed for many common problems, su ...
, statistical theory goes on to consider the balance between having good properties, if tightly defined assumptions hold, and having less good properties that hold under wider conditions.
Background
An "estimator" or "
point estimate
In statistics, point estimation involves the use of sample data to calculate a single value (known as a point estimate since it identifies a point in some parameter space) which is to serve as a "best guess" or "best estimate" of an unknown popu ...
" is a
statistic (that is, a function of the data) that is used to infer the value of an unknown
parameter
A parameter (), generally, is any characteristic that can help in defining or classifying a particular system (meaning an event, project, object, situation, etc.). That is, a parameter is an element of a system that is useful, or critical, when ...
in a
statistical model. A common way of phrasing it is "the estimator is the method selected to obtain an estimate of an unknown parameter".
The parameter being estimated is sometimes called the ''
estimand An estimand is a quantity that is to be estimated in a statistical analysis. The term is used to more clearly distinguish the target of inference from the method used to obtain an approximation of this target (i.e., the estimator) and the specific v ...
''. It can be either finite-dimensional (in
parametric and
semi-parametric model In statistics, a semiparametric model is a statistical model that has parametric and nonparametric components.
A statistical model is a parameterized family of distributions: \ indexed by a parameter \theta.
* A parametric model is a model in ...
s), or infinite-dimensional (
semi-parametric and
non-parametric model
Nonparametric statistics is the branch of statistics that is not based solely on parametrized families of probability distributions (common examples of parameters are the mean and variance). Nonparametric statistics is based on either being distri ...
s). If the parameter is denoted
then the estimator is traditionally written by adding a
circumflex over the symbol:
. Being a function of the data, the estimator is itself a
random variable; a particular realization of this random variable is called the "estimate". Sometimes the words "estimator" and "estimate" are used interchangeably.
The definition places virtually no restrictions on which functions of the data can be called the "estimators". The attractiveness of different estimators can be judged by looking at their properties, such as
unbiasedness
In statistics, the bias of an estimator (or bias function) is the difference between this estimator's expected value and the true value of the parameter being estimated. An estimator or decision rule with zero bias is called ''unbiased''. In stat ...
,
mean square error,
consistency
In classical deductive logic, a consistent theory is one that does not lead to a logical contradiction. The lack of contradiction can be defined in either semantic or syntactic terms. The semantic definition states that a theory is consistent ...
,
asymptotic distribution
In mathematics and statistics, an asymptotic distribution is a probability distribution that is in a sense the "limiting" distribution of a sequence of distributions. One of the main uses of the idea of an asymptotic distribution is in providing ...
, etc. The construction and comparison of estimators are the subjects of the
estimation theory. In the context of
decision theory
Decision theory (or the theory of choice; not to be confused with choice theory) is a branch of applied probability theory concerned with the theory of making decisions based on assigning probabilities to various factors and assigning numerical ...
, an estimator is a type of
decision rule
In decision theory, a decision rule is a function which maps an observation to an appropriate action. Decision rules play an important role in the theory of statistics and economics, and are closely related to the concept of a strategy in game t ...
, and its performance may be evaluated through the use of
loss functions.
When the word "estimator" is used without a qualifier, it usually refers to point estimation. The estimate in this case is a single point in the
parameter space The parameter space is the space of possible parameter values that define a particular mathematical model, often a subset of finite-dimensional Euclidean space. Often the parameters are inputs of a function, in which case the technical term for th ...
. There also exists another type of estimator:
interval estimators, where the estimates are subsets of the parameter space.
The problem of
density estimation
In statistics, probability density estimation or simply density estimation is the construction of an estimate, based on observed data, of an unobservable underlying probability density function. The unobservable density function is thought of ...
arises in two applications. Firstly, in estimating the
probability density function
In probability theory, a probability density function (PDF), or density of a continuous random variable, is a function whose value at any given sample (or point) in the sample space (the set of possible values taken by the random variable) ca ...
s of random variables and secondly in estimating the
spectral density function of a
time series
In mathematics, a time series is a series of data points indexed (or listed or graphed) in time order. Most commonly, a time series is a sequence taken at successive equally spaced points in time. Thus it is a sequence of discrete-time data. Ex ...
. In these problems the estimates are functions that can be thought of as point estimates in an infinite dimensional space, and there are corresponding interval estimation problems.
Definition
Suppose a fixed ''parameter''
needs to be estimated. Then an "estimator" is a function that maps the
sample space
In probability theory, the sample space (also called sample description space, possibility space, or outcome space) of an experiment or random trial is the set of all possible outcomes or results of that experiment. A sample space is usually den ...
to a set of ''sample estimates''. An estimator of
is usually denoted by the symbol
. It is often convenient to express the theory using the
algebra of random variables
The algebra of random variables in statistics, provides rules for the symbolic manipulation of random variables, while avoiding delving too deeply into the mathematically sophisticated ideas of probability theory. Its symbolism allows the treat ...
: thus if ''X'' is used to denote a
random variable corresponding to the observed data, the estimator (itself treated as a random variable) is symbolised as a function of that random variable,
. The estimate for a particular observed data value
(i.e. for
) is then
, which is a fixed value. Often an abbreviated notation is used in which
is interpreted directly as a
random variable, but this can cause confusion.
Quantified properties
The following definitions and attributes are relevant.
Error
For a given sample
, the "
error
An error (from the Latin ''error'', meaning "wandering") is an action which is inaccurate or incorrect. In some usages, an error is synonymous with a mistake. The etymology derives from the Latin term 'errare', meaning 'to stray'.
In statistics ...
" of the estimator
is defined as
:
where
is the parameter being estimated. The error, ''e'', depends not only on the estimator (the estimation formula or procedure), but also on the sample.
Mean squared error
The
mean squared error
In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference between ...
of
is defined as the expected value (probability-weighted average, over all samples) of the squared errors; that is,
:
It is used to indicate how far, on average, the collection of estimates are from the single parameter being estimated. Consider the following analogy. Suppose the parameter is the bull's-eye of a target, the estimator is the process of shooting arrows at the target, and the individual arrows are estimates (samples). Then high MSE means the average distance of the arrows from the bull's eye is high, and low MSE means the average distance from the bull's eye is low. The arrows may or may not be clustered. For example, even if all arrows hit the same point, yet grossly miss the target, the MSE is still relatively large. However, if the MSE is relatively low then the arrows are likely more highly clustered (than highly dispersed) around the target.
Sampling deviation
For a given sample
, the ''sampling deviation'' of the estimator
is defined as
:
where
is the
expected value of the estimator. The sampling deviation, ''d'', depends not only on the estimator, but also on the sample.
Variance
The
variance
In probability theory and statistics, variance is the expectation of the squared deviation of a random variable from its population mean or sample mean. Variance is a measure of dispersion, meaning it is a measure of how far a set of numbe ...
of
is the expected value of the squared sampling deviations; that is,