Model Evidence

A marginal likelihood is a likelihood function that has been integrated over the parameter space. In Bayesian statistics, it represents the probability of generating the observed sample from a prior and is therefore often referred to as model evidence or simply evidence.

Concept

Given a set of independent identically distributed data points $\mathbf=(x_1,\ldots,x_n),$ where $x_i \sim p(x, \theta)$ according to some probability distribution parameterized by $\theta$, where $\theta$ itself is a random variable described by a distribution, i.e. $\theta \sim p(\theta\mid\alpha),$ the marginal likelihood in general asks what the probability $p(\mathbf\mid\alpha)$ is, where $\theta$ has been marginalized out (integrated out):

:$p(\mathbf\mid\alpha) = \int_\theta p(\mathbf\mid\theta) \, p(\theta\mid\alpha)\ \operatorname\!\theta$

The above definition is phrased in the context of Bayesian statistics in which case $p(\theta\mid\alpha)$ is called prior density and $p(\mathbf\mid\theta)$ is the likelihood. The marginal likelihood quantifies the agreement between data and prior in a geometric sense made precise in de Carvalho et al. (2019). In classical (frequentist) statistics, the concept of marginal likelihood occurs instead in the context of a joint parameter $\theta = (\psi,\lambda)$, where $\psi$ is the actual parameter of interest, and $\lambda$ is a non-interesting nuisance parameter. If there exists a probability distribution for $\lambda$, it is often desirable to consider the likelihood function only in terms of $\psi$, by marginalizing out $\lambda$:
:$\mathcal(\psi;\mathbf) = p(\mathbf\mid\psi) = \int_\lambda p(\mathbf\mid\lambda,\psi) \, p(\lambda\mid\psi) \ \operatorname\!\lambda$

Unfortunately, marginal likelihoods are generally difficult to compute. Exact solutions are known for a small class of distributions, particularly when the marginalized-out parameter is the conjugate prior of the distribution of the data. In other cases, some kind of numerical integration method is needed, either a general method such as Gaussian integration or a Monte Carlo method, or a method specialized to statistical problems such as the Laplace approximation, Gibbs/Metropolis sampling, or the EM algorithm.

It is also possible to apply the above considerations to a single random variable (data point) $x$, rather than a set of observations. In a Bayesian context, this is equivalent to the prior predictive distribution of a data point.

Applications

Bayesian model comparison

In Bayesian model comparison, the marginalized variables $\theta$ are parameters for a particular type of model, and the remaining variable $M$ is the identity of the model itself. In this case, the marginalized likelihood is the probability of the data given the model type, not assuming any particular model parameters. Writing $\theta$ for the model parameters, the marginal likelihood for the model ''M'' is

:$p(\mathbf\mid M) = \int p(\mathbf\mid\theta, M) \, p(\theta\mid M) \, \operatorname\!\theta$

It is in this context that the term ''model evidence'' is normally used. This quantity is important because the posterior odds ratio for a model ''M''₁ against another model ''M''₂ involves a ratio of marginal likelihoods, the so-called Bayes factor:

:$\frac = \frac \, \frac$

which can be stated schematically as

:posterior odds = prior odds × Bayes factor

See also

* Empirical Bayes methods
* Lindley's paradox
* Marginal probability
* Bayesian information criterion

References

* Charles S. Bos. "A comparison of marginal likelihood computation methods". In W. Härdle and B. Ronz, editors, ''COMPSTAT 2002: Proceedings in Computational Statistics'', pp. 111–117. 2002. ''(Available as a preprint on the web

''
* de Carvalho, Miguel; Page, Garritt; Barney, Bradley (2019). "On the geometry of Bayesian inference". ''Bayesian Analysis''. 14 (4): 1013‒1036. ''(Available as a preprint on the web

''
* {{cite book , first=Ben , last=Lambert , chapter=The devil is in the denominator , pages=109–120 , title=A Student's Guide to Bayesian Statistics , location= , publisher=Sage , year=2018 , isbn=978-1-4739-1636-4

The on-line textbook: Information Theory, Inference, and Learning Algorithms
by David J.C. MacKay.

Bayesian statistics