In
estimation theory
Estimation theory is a branch of statistics that deals with estimating the values of parameters based on measured empirical data that has a random component. The parameters describe an underlying physical setting in such a way that their valu ...
and
statistics
Statistics (from German language, German: ''wikt:Statistik#German, Statistik'', "description of a State (polity), state, a country") is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of ...
, the Cramér–Rao bound (CRB) expresses a lower bound on the
variance
In probability theory and statistics, variance is the expectation of the squared deviation of a random variable from its population mean or sample mean. Variance is a measure of dispersion, meaning it is a measure of how far a set of numbers ...
of
unbiased estimator
In statistics, the bias of an estimator (or bias function) is the difference between this estimator's expected value and the true value of the parameter being estimated. An estimator or decision rule with zero bias is called ''unbiased''. In stat ...
s of a deterministic (fixed, though unknown) parameter, the variance of any such estimator is at least as high as the inverse of the
Fisher information
In mathematical statistics, the Fisher information (sometimes simply called information) is a way of measuring the amount of information that an observable random variable ''X'' carries about an unknown parameter ''θ'' of a distribution that model ...
. Equivalently, it expresses an upper bound on the
precision
Precision, precise or precisely may refer to:
Science, and technology, and mathematics Mathematics and computing (general)
* Accuracy and precision, measurement deviation from true value and its scatter
* Significant figures, the number of digit ...
(the inverse of variance) of unbiased estimators: the precision of any such estimator is at most the Fisher information.
The result is named in honor of
Harald Cramér
Harald Cramér (; 25 September 1893 – 5 October 1985) was a Swedish mathematician, actuary, and statistician, specializing in mathematical statistics and probabilistic number theory. John Kingman described him as "one of the giants of statist ...
and
C. R. Rao
Calyampudi Radhakrishna Rao FRS (born 10 September 1920), commonly known as C. R. Rao, is an Indian-American mathematician and statistician. He is currently professor emeritus at Pennsylvania State University and Research Professor at the Un ...
,
but has independently also been derived by
Maurice Fréchet Maurice may refer to:
People
*Saint Maurice (died 287), Roman legionary and Christian martyr
*Maurice (emperor) or Flavius Mauricius Tiberius Augustus (539–602), Byzantine emperor
* Maurice (bishop of London) (died 1107), Lord Chancellor and L ...
,
Georges Darmois
Georges Darmois (24 June 1888 – 3 January 1960) was a French mathematician and statistician. He pioneered in the theory of sufficiency, in stellar statistics, and in factor analysis. He was also one of the first French mathematicians to teach ...
,
as well as
Alexander Aitken and
Harold Silverstone.
An unbiased estimator that achieves this lower bound is said to be (fully) ''
efficient''. Such a solution achieves the lowest possible
mean squared error
In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference between ...
among all unbiased methods, and is therefore the
minimum variance unbiased (MVU) estimator. However, in some cases, no unbiased technique exists which achieves the bound. This may occur either if for any unbiased estimator, there exists another with a strictly smaller variance, or if an MVU estimator exists, but its variance is strictly greater than the inverse of the Fisher information.
The Cramér–Rao bound can also be used to bound the variance of estimators of given bias. In some cases, a biased approach can result in both a variance and a
mean squared error
In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference between ...
that are the unbiased Cramér–Rao lower bound; see
estimator bias
In statistics, the bias of an estimator (or bias function) is the difference between this estimator's expected value and the true value of the parameter being estimated. An estimator or decision rule with zero bias is called ''unbiased''. In stat ...
.
Statement
The Cramér–Rao bound is stated in this section for several increasingly general cases, beginning with the case in which the parameter is a
scalar
Scalar may refer to:
*Scalar (mathematics), an element of a field, which is used to define a vector space, usually the field of real numbers
* Scalar (physics), a physical quantity that can be described by a single element of a number field such ...
and its estimator is
unbiased
Bias is a disproportionate weight ''in favor of'' or ''against'' an idea or thing, usually in a way that is closed-minded, prejudicial, or unfair. Biases can be innate or learned. People may develop biases for or against an individual, a group, ...
. All versions of the bound require certain regularity conditions, which hold for most well-behaved distributions. These conditions are listed
later in this section.
Scalar unbiased case
Suppose
is an unknown deterministic parameter that is to be estimated from
independent observations (measurements) of
, each from a distribution according to some
probability density function
In probability theory, a probability density function (PDF), or density of a continuous random variable, is a function whose value at any given sample (or point) in the sample space (the set of possible values taken by the random variable) can ...
. The
variance
In probability theory and statistics, variance is the expectation of the squared deviation of a random variable from its population mean or sample mean. Variance is a measure of dispersion, meaning it is a measure of how far a set of numbers ...
of any ''unbiased'' estimator
of
is then bounded by the
reciprocal
Reciprocal may refer to:
In mathematics
* Multiplicative inverse, in mathematics, the number 1/''x'', which multiplied by ''x'' gives the product 1, also known as a ''reciprocal''
* Reciprocal polynomial, a polynomial obtained from another pol ...
of the
Fisher information
In mathematical statistics, the Fisher information (sometimes simply called information) is a way of measuring the amount of information that an observable random variable ''X'' carries about an unknown parameter ''θ'' of a distribution that model ...
:
:
where the Fisher information
is defined by
:
and
is the
natural logarithm
The natural logarithm of a number is its logarithm to the base of the mathematical constant , which is an irrational and transcendental number approximately equal to . The natural logarithm of is generally written as , , or sometimes, if ...
of the
likelihood function
The likelihood function (often simply called the likelihood) represents the probability of random variable realizations conditional on particular values of the statistical parameters. Thus, when evaluated on a given sample, the likelihood funct ...
for a single sample
and
denotes the
expected value
In probability theory, the expected value (also called expectation, expectancy, mathematical expectation, mean, average, or first moment) is a generalization of the weighted average. Informally, the expected value is the arithmetic mean of a l ...
with respect to the density
of
. If
is twice differentiable and certain regularity conditions hold, then the Fisher information can also be defined as follows:
:
The
efficiency
Efficiency is the often measurable ability to avoid wasting materials, energy, efforts, money, and time in doing something or in producing a desired result. In a more general sense, it is the ability to do things well, successfully, and without ...
of an unbiased estimator
measures how close this estimator's variance comes to this lower bound; estimator efficiency is defined as
:
or the minimum possible variance for an unbiased estimator divided by its actual variance.
The Cramér–Rao lower bound thus gives
:
.
General scalar case
A more general form of the bound can be obtained by considering a biased estimator
, whose expectation is not
but a function of this parameter, say,
. Hence
is not generally equal to 0. In this case, the bound is given by
:
where
is the derivative of
(by
), and
is the Fisher information defined above.
Bound on the variance of biased estimators
Apart from being a bound on estimators of functions of the parameter, this approach can be used to derive a bound on the variance of biased estimators with a given bias, as follows. Consider an estimator
with bias
, and let
. By the result above, any unbiased estimator whose expectation is
has variance greater than or equal to
. Thus, any estimator
whose bias is given by a function
satisfies
:
The unbiased version of the bound is a special case of this result, with
.
It's trivial to have a small variance − an "estimator" that is constant has a variance of zero. But from the above equation we find that the
mean squared error
In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference between ...
of a biased estimator is bounded by
:
using the standard decomposition of the MSE. Note, however, that if
this bound might be less than the unbiased Cramér–Rao bound
. For instance, in the
example of estimating variance below,
.
Multivariate case
Extending the Cramér–Rao bound to multiple parameters, define a parameter column
vector
Vector most often refers to:
*Euclidean vector, a quantity with a magnitude and a direction
*Vector (epidemiology), an agent that carries and transmits an infectious pathogen into another living organism
Vector may also refer to:
Mathematic ...
:
with probability density function
which satisfies the two
regularity conditions below.
The
Fisher information matrix
In mathematical statistics, the Fisher information (sometimes simply called information) is a way of measuring the amount of information that an observable random variable ''X'' carries about an unknown parameter ''θ'' of a distribution that model ...
is a
matrix with element
defined as
:
Let
be an estimator of any vector function of parameters,
, and denote its expectation vector