In
mathematics
Mathematics is an area of knowledge that includes the topics of numbers, formulas and related structures, shapes and the spaces in which they are contained, and quantities and their changes. These topics are represented in modern mathematics ...
, more specifically in the theory of
Monte Carlo method
Monte Carlo methods, or Monte Carlo experiments, are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results. The underlying concept is to use randomness to solve problems that might be determi ...
s, variance reduction is a procedure used to increase the
precision
Precision, precise or precisely may refer to:
Science, and technology, and mathematics Mathematics and computing (general)
* Accuracy and precision, measurement deviation from true value and its scatter
* Significant figures, the number of digit ...
of the
estimates
{{otheruses, Estimate (disambiguation)
In the Westminster system of government, the ''Estimates'' are an outline of government spending for the following fiscal year presented by the cabinet to parliament. The Estimates are drawn up by bureaucrat ...
obtained for a given simulation or computational effort.
Every output random variable from the simulation is associated with a
variance
In probability theory and statistics, variance is the expectation of the squared deviation of a random variable from its population mean or sample mean. Variance is a measure of dispersion, meaning it is a measure of how far a set of numbers ...
which limits the precision of the simulation results. In order to make a simulation statistically efficient, i.e., to obtain a greater precision and smaller
confidence interval
In frequentist statistics, a confidence interval (CI) is a range of estimates for an unknown parameter. A confidence interval is computed at a designated ''confidence level''; the 95% confidence level is most common, but other levels, such as 9 ...
s for the output random variable of interest, variance reduction techniques can be used. The main ones are common random numbers,
antithetic variates In statistics, the antithetic variates method is a variance reduction technique used in Monte Carlo methods. Considering that the error in the simulated signal (using Monte Carlo methods) has a one-over square root convergence, a very large number ...
,
control variate
The control variates method is a variance reduction technique used in Monte Carlo methods. It exploits information about the errors in estimates of known quantities to reduce the error of an estimate of an unknown quantity.
Glasserman, P. (2004). ...
s,
importance sampling
Importance sampling is a Monte Carlo method for evaluating properties of a particular distribution, while only having samples generated from a different distribution than the distribution of interest. Its introduction in statistics is generally att ...
,
stratified sampling
In statistics, stratified sampling is a method of sampling from a population which can be partitioned into subpopulations.
In statistical surveys, when subpopulations within an overall population vary, it could be advantageous to sample each s ...
,
moment matching
Moment or Moments may refer to:
* Present time
Music
* The Moments, American R&B vocal group Albums
* ''Moment'' (Dark Tranquillity album), 2020
* ''Moment'' (Speed album), 1998
* ''Moments'' (Darude album)
* ''Moments'' (Christine Guldbrand ...
,
conditional Monte Carlo
Conditional (if then) may refer to:
*Causal conditional, if X then Y, where X is a cause of Y
*Conditional probability, the probability of an event A given that another event B has occurred
*Conditional proof, in logic: a proof that asserts a co ...
and
quasi random variables
Quasi (phonetics 'kwa - zee') is an American indie rock band formed in Portland, Oregon in 1993 by former spouses Sam Coomes ( vocals, guitar, rocksichord, various keyboards, bass) and Janet Weiss (vocals and drums). Joanna Bolme performed a ...
. For simulation with
black-box
In science, computing, and engineering, a black box is a system which can be viewed in terms of its inputs and outputs (or transfer characteristics), without any knowledge of its internal workings. Its implementation is "opaque" (black). The te ...
models
subset simulation Subset simulation is a method used in reliability engineering to compute small (i.e., rare event) failure probabilities encountered in engineering systems. The basic idea is to express a small failure probability as a product of larger conditional ...
and
line sampling can also be used. Under these headings are a variety of specialized techniques; for example, particle transport simulations make extensive use of "weight windows" and "splitting/Russian roulette" techniques, which are a form of importance sampling.
Crude Monte Carlo simulation
Suppose one wants to compute
with the random variable
defined on the
probability space
In probability theory, a probability space or a probability triple (\Omega, \mathcal, P) is a mathematical construct that provides a formal model of a random process or "experiment". For example, one can define a probability space which models t ...
. Monte Carlo does this by sampling
i.i.d. copies
of
and then to estimate
via the sample-mean estimator
:
Under further mild conditions such as
, a
central limit theorem
In probability theory, the central limit theorem (CLT) establishes that, in many situations, when independent random variables are summed up, their properly normalized sum tends toward a normal distribution even if the original variables themselv ...
will apply such that for large
, the distribution of
converges to a normal distribution with mean
and standard error
. Because the standard deviation only converges towards
at the rate
, implying one needs to increase the number of simulations (
) by a factor of
to halve the standard deviation of
, variance reduction methods are often useful for obtaining more precise estimates for
without needing very large numbers of simulations.
Common Random Numbers (CRN)
The common random numbers variance reduction technique is a popular and useful variance reduction technique which applies when we are comparing two or more alternative configurations (of a system) instead of investigating a single configuration. CRN has also been called ''correlated sampling'', ''matched streams'' or ''matched pairs''.
CRN requires synchronization of the random number streams, which ensures that in addition to using the same random numbers to simulate all configurations, a specific random number used for a specific purpose in one configuration is used for exactly the same purpose in all other configurations. For example, in queueing theory, if we are comparing two different configurations of tellers in a bank, we would want the (random) time of arrival of the ''N-''th customer to be generated using the same draw from a random number stream for both configurations.
Underlying principle of the CRN technique
Suppose
and
are the observations from the first and second configurations on the ''j-''th independent replication.
We want to estimate
:
If we perform ''n'' replications of each configuration and let
:
then
and
is an unbiased estimator of
.
And since the
's are independent identically distributed random variables,
:
In case of independent sampling, i.e., no common random numbers used then Cov(''X''
1''j'', ''X''
2''j'') = 0. But if we succeed to induce an element of positive correlation between ''X''
1 and ''X''
2 such that Cov(''X''
1''j'', ''X''
2''j'') > 0, it can be seen from the equation above that the variance is reduced.
It can also be observed that if the CRN induces a negative correlation, i.e., Cov(''X''
1''j'', ''X''
2''j'') < 0, this technique can actually backfire, where the variance is increased and not decreased (as intended).
See also
*
Explained variance In statistics, explained variation measures the proportion to which a mathematical model accounts for the variation (dispersion) of a given data set. Often, variation is quantified as variance; then, the more specific term explained variance can be ...
References
*
*{{cite journal , last=Kahn , first=H. , last2=Marshall , first2=A. W. , year=1953 , title=Methods of Reducing Sample Size in Monte Carlo Computations , journal=
Journal of the Operations Research Society of America
''Operations Research'' is a bimonthly peer-reviewed academic journal covering operations research that is published by the Institute for Operations Research and the Management Sciences. It was established in 1952 as the ''Journal of the Operation ...
, volume=1 , issue=5 , pages=263–271 , doi=10.1287/opre.1.5.263
*MCNP — A General Monte Carlo N-Particle Transport Code, Version 5 Los Alamos Report LA-UR-03-1987
Monte Carlo methods