HOME

TheInfoList



OR:

In
Bayesian statistics Bayesian statistics is a theory in the field of statistics based on the Bayesian interpretation of probability where probability expresses a ''degree of belief'' in an event. The degree of belief may be based on prior knowledge about the event, ...
, a hyperparameter is a parameter of a
prior distribution In Bayesian statistical inference, a prior probability distribution, often simply called the prior, of an uncertain quantity is the probability distribution that would express one's beliefs about this quantity before some evidence is taken into ...
; the term is used to distinguish them from parameters of the model for the underlying system under analysis. For example, if one is using a
beta distribution In probability theory and statistics, the beta distribution is a family of continuous probability distributions defined on the interval , 1in terms of two positive parameters, denoted by ''alpha'' (''α'') and ''beta'' (''β''), that appear as ...
to model the distribution of the parameter ''p'' of a
Bernoulli distribution In probability theory and statistics, the Bernoulli distribution, named after Swiss mathematician Jacob Bernoulli,James Victor Uspensky: ''Introduction to Mathematical Probability'', McGraw-Hill, New York 1937, page 45 is the discrete probabi ...
, then: * ''p'' is a parameter of the underlying system (Bernoulli distribution), and * ''α'' and ''β'' are parameters of the prior distribution (beta distribution), hence ''hyper''parameters. One may take a single value for a given hyperparameter, or one can iterate and take a probability distribution on the hyperparameter itself, called a hyperprior.


Purpose

One often uses a prior which comes from a
parametric family In mathematics and its applications, a parametric family or a parameterized family is a family of objects (a set of related objects) whose differences depend only on the chosen values for a set of parameters. Common examples are parametrized (fam ...
of probability distributions – this is done partly for explicitness (so one can write down a distribution, and choose the form by varying the hyperparameter, rather than trying to produce an arbitrary function), and partly so that one can ''vary'' the hyperparameter, particularly in the method of ''
conjugate prior In Bayesian probability theory, if the posterior distribution p(\theta \mid x) is in the same probability distribution family as the prior probability distribution p(\theta), the prior and posterior are then called conjugate distributions, and ...
s,'' or for ''sensitivity analysis.''


Conjugate priors

When using a conjugate prior, the posterior distribution will be from the same family, but will have different hyperparameters, which reflect the added information from the data: in subjective terms, one's beliefs have been updated. For a general prior distribution, this is computationally very involved, and the posterior may have an unusual or hard to describe form, but with a conjugate prior, there is generally a simple formula relating the values of the hyperparameters of the posterior to those of the prior, and thus the computation of the posterior distribution is very easy.


Sensitivity analysis

A key concern of users of Bayesian statistics, and criticism by critics, is the dependence of the posterior distribution on one's prior. Hyperparameters address this by allowing one to easily vary them and see how the posterior distribution (and various statistics of it, such as
credible intervals In Bayesian statistics, a credible interval is an interval within which an unobserved parameter value falls with a particular probability. It is an interval in the domain of a posterior probability distribution or a predictive distribution. Th ...
) vary: one can see how ''sensitive'' one's conclusions are to one's prior assumptions, and the process is called ''
sensitivity analysis Sensitivity analysis is the study of how the uncertainty in the output of a mathematical model or system (numerical or otherwise) can be divided and allocated to different sources of uncertainty in its inputs. A related practice is uncertainty anal ...
.'' Similarly, one may use a prior distribution with a range for a hyperparameter, perhaps reflecting uncertainty in the correct prior to take, and reflect this in a range for final uncertainty.Giulio D'Agostini
Purely subjective assessment of prior probabilities
i

/ref>


Hyperpriors

Instead of using a single value for a given hyperparameter, one can instead consider a probability distribution of the hyperparameter itself; this is called a " hyperprior." In principle, one may iterate this, calling parameters of a hyperprior "hyperhyperparameters," and so forth.


See also

*
Empirical Bayes method Empirical Bayes methods are procedures for statistical inference in which the prior probability distribution is estimated from the data. This approach stands in contrast to standard Bayesian methods, for which the prior distribution is fixed b ...


References


Further reading

* * * {{cite book , last=Kruschke , first=J. K. , author-link=John K. Kruschke , year=2010 , title=Doing Bayesian Data Analysis: A Tutorial with R and BUGS , publisher=Academic Press , isbn=978-0-12-381485-2 , pages=241–264 , url=https://books.google.com/books?id=ZRMJ-CebFm4C&pg=PA241 Bayesian statistics Sensitivity analysis