Universal Estimator

In statistics, an estimator is a rule for calculating an estimate of a given quantity based on observed data: thus the rule (the estimator), the quantity of interest (the estimand) and its result (the estimate) are distinguished. For example, the sample mean is a commonly used estimator of the population mean.

There are point and interval estimators. The point estimators yield single-valued results. This is in contrast to an interval estimator, where the result would be a range of plausible values. "Single value" does not necessarily mean "single number", but includes vector valued or function valued estimators.

''Estimation theory'' is concerned with the properties of estimators; that is, with defining properties that can be used to compare different estimators (different rules for creating estimates) for the same quantity, based on the same data. Such properties can be used to determine the best rules to use under given circumstances. However, in robust statistics, statistical theory goes on to consider the balance between having good properties, if tightly defined assumptions hold, and having less good properties that hold under wider conditions.

Background

An "estimator" or "point estimate" is a statistic (that is, a function of the data) that is used to infer the value of an unknown parameter in a statistical model. A common way of phrasing it is "the estimator is the method selected to obtain an estimate of an unknown parameter".
The parameter being estimated is sometimes called the '' estimand''. It can be either finite-dimensional (in parametric and semi-parametric models), or infinite-dimensional ( semi-parametric and non-parametric models). If the parameter is denoted $\theta$ then the estimator is traditionally written by adding a circumflex over the symbol: $\widehat$. Being a function of the data, the estimator is itself a random variable; a particular realization of this random variable is called the "estimate". Sometimes the words "estimator" and "estimate" are used interchangeably.

The definition places virtually no restrictions on which functions of the data can be called the "estimators". The attractiveness of different estimators can be judged by looking at their properties, such as unbiasedness, mean square error, consistency, asymptotic distribution, etc. The construction and comparison of estimators are the subjects of the estimation theory. In the context of decision theory, an estimator is a type of decision rule, and its performance may be evaluated through the use of loss functions.

When the word "estimator" is used without a qualifier, it usually refers to point estimation. The estimate in this case is a single point in the parameter space. There also exists another type of estimator: interval estimators, where the estimates are subsets of the parameter space.

The problem of density estimation arises in two applications. Firstly, in estimating the probability density functions of random variables and secondly in estimating the spectral density function of a time series. In these problems the estimates are functions that can be thought of as point estimates in an infinite dimensional space, and there are corresponding interval estimation problems.

Definition

Suppose a fixed ''parameter'' $\theta$ needs to be estimated. Then an "estimator" is a function that maps the sample space to a set of ''sample estimates''. An estimator of $\theta$ is usually denoted by the symbol $\widehat$. It is often convenient to express the theory using the algebra of random variables: thus if ''X'' is used to denote a random variable corresponding to the observed data, the estimator (itself treated as a random variable) is symbolised as a function of that random variable, $\widehat(X)$. The estimate for a particular observed data value $x$ (i.e. for $X=x$) is then $\widehat(x)$, which is a fixed value. Often an abbreviated notation is used in which $\widehat$ is interpreted directly as a random variable, but this can cause confusion.

Quantified properties

The following definitions and attributes are relevant.

Error

For a given sample $x$, the " error" of the estimator $\widehat$ is defined as
:$e(x)=\widehat(x) - \theta,$
where $\theta$ is the parameter being estimated. The error, ''e'', depends not only on the estimator (the estimation formula or procedure), but also on the sample.

Mean squared error

The mean squared error of $\widehat$ is defined as the expected value (probability-weighted average, over all samples) of the squared errors; that is,
:\operatorname(\widehat) = \operatorname \widehat(X) - \theta)^2
It is used to indicate how far, on average, the collection of estimates are from the single parameter being estimated. Consider the following analogy. Suppose the parameter is the bull's-eye of a target, the estimator is the process of shooting arrows at the target, and the individual arrows are estimates (samples). Then high MSE means the average distance of the arrows from the bull's eye is high, and low MSE means the average distance from the bull's eye is low. The arrows may or may not be clustered. For example, even if all arrows hit the same point, yet grossly miss the target, the MSE is still relatively large. However, if the MSE is relatively low then the arrows are likely more highly clustered (than highly dispersed) around the target.

Sampling deviation

For a given sample $x$, the ''sampling deviation'' of the estimator $\widehat$ is defined as
:$d(x) =\widehat(x) - \operatorname( \widehat(X) ) =\widehat(x) - \operatorname( \widehat ),$
where $\operatorname( \widehat(X) )$ is the expected value of the estimator. The sampling deviation, ''d'', depends not only on the estimator, but also on the sample.

Variance

The variance of $\widehat$ is the expected value of the squared sampling deviations; that is, \operatorname(\widehat) = \operatorname \widehat - \operatorname[\widehat ^2">widehat.html" ;"title="\widehat - \operatorname[\widehat">\widehat - \operatorname[\widehat ^2/math>. It is used to indicate how far, on average, the collection of estimates are from the ''expected value'' of the estimates. (Note the difference between MSE and variance.) If the parameter is the bull's-eye of a target, and the arrows are estimates, then a relatively high variance means the arrows are dispersed, and a relatively low variance means the arrows are clustered. Even if the variance is low, the cluster of arrows may still be far off-target, and even if the variance is high, the diffuse collection of arrows may still be unbiased. Finally, even if all arrows grossly miss the target, if they nevertheless all hit the same point, the variance is zero.

Bias

The bias of an estimator">bias