In
statistics
Statistics (from German language, German: ', "description of a State (polity), state, a country") is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of data. In applying statistics to a s ...
, the Rao–Blackwell theorem, sometimes referred to as the Rao–Blackwell–Kolmogorov theorem, is a result that characterizes the transformation of an arbitrarily crude
estimator
In statistics, an estimator is a rule for calculating an estimate of a given quantity based on Sample (statistics), observed data: thus the rule (the estimator), the quantity of interest (the estimand) and its result (the estimate) are distinguish ...
into an estimator that is optimal by the
mean-squared-error criterion or any of a variety of similar criteria.
The Rao–Blackwell theorem states that if ''g''(''X'') is any kind of
estimator
In statistics, an estimator is a rule for calculating an estimate of a given quantity based on Sample (statistics), observed data: thus the rule (the estimator), the quantity of interest (the estimand) and its result (the estimate) are distinguish ...
of a parameter θ, then the
conditional expectation
In probability theory, the conditional expectation, conditional expected value, or conditional mean of a random variable is its expected value evaluated with respect to the conditional probability distribution. If the random variable can take on ...
of ''g''(''X'') given ''T''(''X''), where ''T'' is a
sufficient statistic
In statistics, sufficiency is a property of a statistic computed on a sample dataset in relation to a parametric model of the dataset. A sufficient statistic contains all of the information that the dataset provides about the model parameters. It ...
, is typically a better estimator of θ, and is never worse. Sometimes one can very easily construct a very crude estimator ''g''(''X''), and then evaluate that conditional expected value to get an estimator that is in various senses optimal.
The theorem is named after
C.R. Rao and
David Blackwell. The process of transforming an estimator using the Rao–Blackwell theorem can be referred to as Rao–Blackwellization. The transformed
estimator
In statistics, an estimator is a rule for calculating an estimate of a given quantity based on Sample (statistics), observed data: thus the rule (the estimator), the quantity of interest (the estimand) and its result (the estimate) are distinguish ...
is called the Rao–Blackwell estimator.
[
]
Definitions
*An estimator
In statistics, an estimator is a rule for calculating an estimate of a given quantity based on Sample (statistics), observed data: thus the rule (the estimator), the quantity of interest (the estimand) and its result (the estimate) are distinguish ...
δ(''X'') is an ''observable'' random variable (i.e. a statistic
A statistic (singular) or sample statistic is any quantity computed from values in a sample which is considered for a statistical purpose. Statistical purposes include estimating a population parameter, describing a sample, or evaluating a hypot ...
) used for estimating some ''unobservable
An unobservable (also called impalpable) is an entity whose existence, nature, properties, qualities or relations are not directly observable by humans. In philosophy of science, typical examples of "unobservables" are the force of gravity, causa ...
'' quantity. For example, one may be unable to observe the average height of ''all'' male students at some university, but one may observe the heights of a random sample of 40 of them. The average height of those 40—the "sample average"—may be used as an estimator of the unobservable "population average".
*A sufficient statistic
In statistics, sufficiency is a property of a statistic computed on a sample dataset in relation to a parametric model of the dataset. A sufficient statistic contains all of the information that the dataset provides about the model parameters. It ...
''T''(''X'') is a statistic calculated from data ''X'' to estimate some parameter θ for which no other statistic which can be calculated from data X provides any additional information about θ. It is defined as an ''observable'' random variable
A random variable (also called random quantity, aleatory variable, or stochastic variable) is a Mathematics, mathematical formalization of a quantity or object which depends on randomness, random events. The term 'random variable' in its mathema ...
such that the conditional probability
In probability theory, conditional probability is a measure of the probability of an Event (probability theory), event occurring, given that another event (by assumption, presumption, assertion or evidence) is already known to have occurred. This ...
distribution of all observable data ''X'' given ''T''(''X'') does not depend on the ''unobservable'' parameter θ, such as the mean or standard deviation of the whole population from which the data ''X'' was taken. In the most frequently cited examples, the "unobservable" quantities are parameters that parametrize a known family of probability distribution
In probability theory and statistics, a probability distribution is a Function (mathematics), function that gives the probabilities of occurrence of possible events for an Experiment (probability theory), experiment. It is a mathematical descri ...
s according to which the data are distributed.
::In other words, a sufficient statistic
In statistics, sufficiency is a property of a statistic computed on a sample dataset in relation to a parametric model of the dataset. A sufficient statistic contains all of the information that the dataset provides about the model parameters. It ...
''T(X)'' for a parameter θ is a statistic
A statistic (singular) or sample statistic is any quantity computed from values in a sample which is considered for a statistical purpose. Statistical purposes include estimating a population parameter, describing a sample, or evaluating a hypot ...
such that the conditional probability of the data ''X'', given ''T''(''X''), does not depend on the parameter θ.
*A Rao–Blackwell estimator δ1(''X'') of an unobservable quantity θ is the conditional expected value E(δ(''X'') , ''T''(''X'')) of some estimator δ(''X'') given a sufficient statistic ''T''(''X''). Call δ(''X'') the "original estimator" and δ1(''X'') the "improved estimator". It is important that the improved estimator be ''observable'', i.e. that it does not depend on θ. Generally, the conditional expected value of one function of these data given another function of these data ''does'' depend on θ, but the very definition of sufficiency given above entails that this one does not.
*The ''mean squared error
In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference betwee ...
'' of an estimator is the expected value of the square of its deviation from the unobservable quantity being estimated of θ.
The theorem
Mean-squared-error version
One case of Rao–Blackwell theorem states:
:The mean squared error of the Rao–Blackwell estimator does not exceed that of the original estimator.
In other words,
:
The essential tools of the proof besides the definition above are the law of total expectation
The proposition in probability theory known as the law of total expectation, the law of iterated expectations (LIE), Adam's law, the tower rule, and the smoothing property of conditional expectation, among other names, states that if X is a random ...
and the fact that for any random variable ''Y'', E(''Y''2) cannot be less than (''Y'')sup>2. That inequality is a case of Jensen's inequality, although it may also be shown to follow instantly from the frequently mentioned fact that
:
More precisely, the mean square error of the Rao-Blackwell estimator has the following decomposition
: