Expectation propagation (EP) is a technique in
Bayesian machine learning.
EP finds approximations to a
probability distribution
In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomenon i ...
.
It uses an
iterative
Iteration is the repetition of a process in order to generate a (possibly unbounded) sequence of outcomes. Each repetition of the process is a single iteration, and the outcome of each iteration is then the starting point of the next iteration. ...
approach that uses the factorization structure of the target distribution.
It differs from other Bayesian approximation approaches such as
variational Bayesian methods
Variational Bayesian methods are a family of techniques for approximating intractable integrals arising in Bayesian inference and machine learning. They are typically used in complex statistical models consisting of observed variables (usually ...
.
More specifically, suppose we wish to approximate an intractable probability distribution
with a tractable distribution
. Expectation propagation achieves this approximation by minimizing the
Kullback-Leibler divergence .
Variational Bayesian methods minimize
instead.
If
is a Gaussian
, then
is minimized with
and
being equal to the
mean
There are several kinds of mean in mathematics, especially in statistics. Each mean serves to summarize a given group of data, often to better understand the overall value (magnitude and sign) of a given data set.
For a data set, the ''arithme ...
of
and the
covariance
In probability theory and statistics, covariance is a measure of the joint variability of two random variables. If the greater values of one variable mainly correspond with the greater values of the other variable, and the same holds for the les ...
of
, respectively; this is called
moment matching
Moment or Moments may refer to:
* Present time
Music
* The Moments, American R&B vocal group Albums
* ''Moment'' (Dark Tranquillity album), 2020
* ''Moment'' (Speed album), 1998
* ''Moments'' (Darude album)
* ''Moments'' (Christine Guldbrand ...
.
Applications
Expectation propagation via moment matching plays a vital role in approximation for
indicator functions that appear when deriving the
message passing equations for
TrueSkill
TrueSkill is a skill-based ranking system developed by Microsoft for use with video game matchmaking on Xbox Live. Unlike the popular Elo rating system, which was initially designed for chess, TrueSkill is designed to support games with more than t ...
.
References
*
External links
Minka's EP papers
Machine learning
Bayesian statistics
{{compsci-stub