HOME

TheInfoList



OR:

The likelihood function (often simply called the likelihood) represents the probability of random variable realizations conditional on particular values of the
statistical parameter In statistics, as opposed to its general use in mathematics, a parameter is any measured quantity of a statistical population that summarises or describes an aspect of the population, such as a mean or a standard deviation. If a population ...
s. Thus, when evaluated on a given sample, the likelihood function indicates which parameter values are more ''likely'' than others, in the sense that they would have made the observed data more probable. Consequently, the likelihood is often written as \mathcal(\theta\mid X) instead of P(X \mid \theta), to emphasize that it is to be understood as a function of the parameters \theta instead of the random variable X. In maximum likelihood estimation, the
arg max In mathematics, the arguments of the maxima (abbreviated arg max or argmax) are the points, or elements, of the domain of some function at which the function values are maximized.For clarity, we refer to the input (''x'') as ''points'' and t ...
of the likelihood function serves as a point estimate for \theta, while local curvature (approximated by the likelihood's Hessian matrix) indicates the estimate's precision. Meanwhile in Bayesian statistics, parameter estimates are derived from the converse of the likelihood, the so-called
posterior probability The posterior probability is a type of conditional probability that results from updating the prior probability with information summarized by the likelihood via an application of Bayes' rule. From an epistemological perspective, the posterior p ...
, which is calculated via
Bayes' rule In probability theory and statistics, Bayes' theorem (alternatively Bayes' law or Bayes' rule), named after Thomas Bayes, describes the probability of an event, based on prior knowledge of conditions that might be related to the event. For exa ...
.


Definition

The likelihood function, parameterized by a (possibly multivariate) parameter \theta, is usually defined differently for discrete and continuous probability distributions (a more general definition is discussed below). Given a probability density or mass function :x\mapsto f(x \mid \theta), \! where x is a realization of the random variable X, the likelihood function is :\theta\mapsto f(x\mid\theta), \! often written :\mathcal(\theta \mid x). \! In other words, when f(x\mid\theta) is viewed as a function of x with \theta fixed, it is a probability density function, and when viewed as a function of \theta with x fixed, it is a likelihood function. The likelihood function does ''not'' specify the probability that \theta is the truth, given the observed sample X = x. Such an interpretation is a common error, with potentially disastrous consequences (see prosecutor's fallacy).


Discrete probability distribution

Let X be a discrete
random variable A random variable (also called random quantity, aleatory variable, or stochastic variable) is a mathematical formalization of a quantity or object which depends on random events. It is a mapping or a function from possible outcomes (e.g., the po ...
with
probability mass function In probability and statistics, a probability mass function is a function that gives the probability that a discrete random variable is exactly equal to some value. Sometimes it is also known as the discrete density function. The probability mass ...
p depending on a parameter \theta. Then the function :\mathcal(\theta \mid x) = p_\theta (x) = P_\theta (X=x), considered as a function of \theta, is the ''likelihood function'', given the outcome x of the random variable X. Sometimes the probability of "the value x of X for the parameter value \theta" is written as or . The likelihood is the probability that a particular outcome x is observed when the true value of the parameter is \theta, equivalent to the probability mass on x; it is ''not'' a probability density over the parameter \theta. The likelihood, \mathcal(\theta \mid x) , should not be confused with P(\theta \mid x), which is the posterior probability of \theta given the data x. Given no event (no data), the likelihood is 1; any non-trivial event will have a lower likelihood.


Example

Consider a simple statistical model of a coin flip: a single parameter p_\text that expresses the "fairness" of the coin. The parameter is the probability that a coin lands heads up ("H") when tossed. p_\text can take on any value within the range 0.0 to 1.0. For a perfectly fair coin, p_\text = 0.5. Imagine flipping a fair coin twice, and observing two heads in two tosses ("HH"). Assuming that each successive coin flip is i.i.d., then the probability of observing HH is :P(\text \mid p_\text=0.5) = 0.5^2 = 0.25. Equivalently, the likelihood at \theta = 0.5 given that "HH" was observed is 0.25: :\mathcal(p_\text=0.5 \mid \text) = 0.25. This is not the same as saying that P(p_\text = 0.5 \mid HH) = 0.25, a conclusion which could only be reached via
Bayes' theorem In probability theory and statistics, Bayes' theorem (alternatively Bayes' law or Bayes' rule), named after Thomas Bayes, describes the probability of an event, based on prior knowledge of conditions that might be related to the event. For examp ...
given knowledge about the marginal probabilities P(p_\text = 0.5) and P(HH). Now suppose that the coin is not a fair coin, but instead that p_\text = 0.3. Then the probability of two heads on two flips is :P(\text \mid p_\text=0.3) = 0.3^2 = 0.09. Hence :\mathcal(p_\text=0.3 \mid \text) = 0.09. More generally, for each value of p_\text, we can calculate the corresponding likelihood. The result of such calculations is displayed in Figure 1. Note that the integral of \mathcal over , 1is 1/3; likelihoods need not integrate or sum to one over the parameter space.


Continuous probability distribution

Let X be a
random variable A random variable (also called random quantity, aleatory variable, or stochastic variable) is a mathematical formalization of a quantity or object which depends on random events. It is a mapping or a function from possible outcomes (e.g., the po ...
following an absolutely continuous probability distribution with
density function In probability theory, a probability density function (PDF), or density of a continuous random variable, is a function whose value at any given sample (or point) in the sample space (the set of possible values taken by the random variable) can ...
f (a function of x) which depends on a parameter \theta. Then the function :\mathcal(\theta \mid x) = f_\theta (x), \, considered as a function of \theta, is the ''likelihood function'' (of \theta, given the outcome X=x). Again, note that \mathcal is not a probability density or mass function over \theta, despite being a function of \theta given the observation X = x.


Relationship between the likelihood and probability density functions

The use of the probability density in specifying the likelihood function above is justified as follows. Given an observation x_j, the likelihood for the interval _j, x_j + h/math>, where h > 0 is a constant, is given by \mathcal(\theta\mid x \in _j, x_j + h . Observe that : \operatorname_\theta \mathcal(\theta\mid x \in _j, x_j + h = \operatorname_\theta \frac \mathcal(\theta\mid x \in _j, x_j + h , since h is positive and constant. Because : \operatorname_\theta \frac 1 h \mathcal(\theta\mid x \in _j, x_j + h = \operatorname_\theta \frac 1 h \Pr(x_j \leq x \leq x_j + h \mid \theta) = \operatorname_\theta \frac 1 h \int_^ f(x\mid \theta) \,dx, where f(x\mid \theta) is the probability density function, it follows that : \operatorname_\theta \mathcal(\theta\mid x \in _j, x_j + h = \operatorname_\theta \frac \int_^ f(x\mid\theta) \,dx . The first
fundamental theorem of calculus The fundamental theorem of calculus is a theorem that links the concept of differentiating a function (calculating its slopes, or rate of change at each time) with the concept of integrating a function (calculating the area under its graph, ...
provides that : \begin & \lim_ \frac 1 h \int_^ f(x\mid\theta) \,dx = f(x_j \mid \theta). \end Then : \begin & \operatorname_\theta \mathcal(\theta\mid x_j) = \operatorname_\theta \left _j,_x_j_+_h_\right.html" ;"title="\lim_ \mathcal(\theta\mid x \in _j, x_j + h \right">\lim_ \mathcal(\theta\mid x \in _j, x_j + h \right\\ pt= & \operatorname_\theta \left \lim_ \frac \int_^ f(x\mid\theta) \,dx \right= \operatorname_\theta f(x_j \mid \theta). \end Therefore, : \operatorname_\theta \mathcal(\theta\mid x_j) = \operatorname_\theta f(x_j \mid \theta), \! and so maximizing the probability density at x_j amounts to maximizing the likelihood of the specific observation x_j .


In general

In measure-theoretic probability theory, the
density function In probability theory, a probability density function (PDF), or density of a continuous random variable, is a function whose value at any given sample (or point) in the sample space (the set of possible values taken by the random variable) can ...
is defined as the Radon–Nikodym derivative of the probability distribution relative to a common dominating measure. The likelihood function is this density interpreted as a function of the parameter, rather than the random variable. Thus, we can construct a likelihood function for any distribution, whether discrete, continuous, a mixture, or otherwise. (Likelihoods are comparable, e.g. for parameter estimation, only if they are Radon–Nikodym derivatives with respect to the same dominating measure.) The above discussion of the likelihood for discrete random variables uses the counting measure, under which the probability density at any outcome equals the probability of that outcome.


Likelihoods for mixed continuous–discrete distributions

The above can be extended in a simple way to allow consideration of distributions which contain both discrete and continuous components. Suppose that the distribution consists of a number of discrete probability masses p_k \theta and a density f(x\mid\theta), where the sum of all the p's added to the integral of f is always one. Assuming that it is possible to distinguish an observation corresponding to one of the discrete probability masses from one which corresponds to the density component, the likelihood function for an observation from the continuous component can be dealt with in the manner shown above. For an observation from the discrete component, the likelihood function for an observation from the discrete component is simply :\mathcal(\theta \mid x )= p_k(\theta), \! where k is the index of the discrete probability mass corresponding to observation x, because maximizing the probability mass (or probability) at x amounts to maximizing the likelihood of the specific observation. The fact that the likelihood function can be defined in a way that includes contributions that are not commensurate (the density and the probability mass) arises from the way in which the likelihood function is defined up to a constant of proportionality, where this "constant" can change with the observation x, but not with the parameter \theta.


Regularity conditions

In the context of parameter estimation, the likelihood function is usually assumed to obey certain conditions, known as regularity conditions. These conditions are in various proofs involving likelihood functions, and need to be verified in each particular application. For maximum likelihood estimation, the existence of a global maximum of the likelihood function is of the utmost importance. By the
extreme value theorem In calculus, the extreme value theorem states that if a real-valued function f is continuous on the closed interval ,b/math>, then f must attain a maximum and a minimum, each at least once. That is, there exist numbers c and d in ,b/math> s ...
, it suffices that the likelihood function is
continuous Continuity or continuous may refer to: Mathematics * Continuity (mathematics), the opposing concept to discreteness; common examples include ** Continuous probability distribution or random variable in probability and statistics ** Continuous g ...
on a
compact Compact as used in politics may refer broadly to a pact or treaty; in more specific cases it may refer to: * Interstate compact * Blood compact, an ancient ritual of the Philippines * Compact government, a type of colonial rule utilized in Britis ...
parameter space for the maximum likelihood estimator to exist. While the continuity assumption is usually met, the compactness assumption about the parameter space is often not, as the bounds of the true parameter values are unknown. In that case,
concavity In calculus, the second derivative, or the second order derivative, of a function is the derivative of the derivative of . Roughly speaking, the second derivative measures how the rate of change of a quantity is itself changing; for example, ...
of the likelihood function plays a key role. More specifically, if the likelihood function is twice continuously differentiable on the k-dimensional parameter space \, \Theta \, assumed to be an
open Open or OPEN may refer to: Music * Open (band), Australian pop/rock band * The Open (band), English indie rock band * Open (Blues Image album), ''Open'' (Blues Image album), 1969 * Open (Gotthard album), ''Open'' (Gotthard album), 1999 * Open (C ...
connected subset of \, \mathbb^ \;, there exists a unique maximum \hat \in \Theta if the matrix of second partials : \mathbf(\theta) \equiv \left , \frac \,\right^ \; is negative definite for every \, \theta \in \Theta \, at which the gradient \; \nabla L \equiv \left , \frac \,\right^ \; vanishes, and if : \lim_ L(\theta) = 0 \;, i.e. the likelihood function approaches a constant on the boundary of the parameter space, \; \partial \Theta \;, which may include the points at infinity if \, \Theta \, is unbounded. Mäkeläinen ''et al''. prove this result using Morse theory while informally appealing to a mountain pass property. Mascarenhas restates their proof using the
mountain pass theorem The mountain pass theorem is an existence theorem from the calculus of variations, originally due to Antonio Ambrosetti and Paul Rabinowitz. Given certain conditions on a function, the theorem demonstrates the existence of a saddle point. The th ...
. In the proofs of consistency and asymptotic normality of the maximum likelihood estimator, additional assumptions are made about the probability densities that form the basis of a particular likelihood function. These conditions were first established by Chanda. In particular, for
almost all In mathematics, the term "almost all" means "all but a negligible amount". More precisely, if X is a set, "almost all elements of X" means "all elements of X but those in a negligible subset of X". The meaning of "negligible" depends on the mathema ...
x, and for all \, \theta \in \Theta \,, :\frac \,, \quad \frac \,, \quad \frac \, exist for all \, r, s, t = 1, 2, \ldots, k \, in order to ensure the existence of a Taylor expansion. Second, for almost all x and for every \, \theta \in \Theta \, it must be that : \left, \frac \ < F_r(x) \,, \quad \left, \frac \ < F_(x) \,, \quad \left, \frac \ < H_(x) where H is such that \, \int_^ H_(z) \mathrmz \leq M < \infty \;. This boundedness of the derivatives is needed to allow for differentiation under the integral sign. And lastly, it is assumed that the information matrix, :\mathbf(\theta) = \int_^ \frac\ \frac\ f\ \mathrmz is
positive definite In mathematics, positive definiteness is a property of any object to which a bilinear form or a sesquilinear form may be naturally associated, which is positive-definite. See, in particular: * Positive-definite bilinear form * Positive-definite fu ...
and \, \left, \mathbf(\theta) \ \, is finite. This ensures that the
score Score or scorer may refer to: *Test score, the result of an exam or test Business * Score Digital, now part of Bauer Radio * Score Entertainment, a former American trading card design and manufacturing company * Score Media, a former Canadian ...
has a finite variance. The above conditions are sufficient, but not necessary. That is, a model that does not meet these regularity conditions may or may not have a maximum likelihood estimator of the properties mentioned above. Further, in case of non-independently or non-identically distributed observations additional properties may need to be assumed. In Bayesian statistics, almost identical regularity conditions are imposed on the likelihood function in order to proof asymptotic normality of the
posterior probability The posterior probability is a type of conditional probability that results from updating the prior probability with information summarized by the likelihood via an application of Bayes' rule. From an epistemological perspective, the posterior p ...
, and therefore to justify a
Laplace approximation In mathematics, Laplace's approximation fits an un-normalised Gaussian approximation to a (twice differentiable) un-normalised target density. In Bayesian statistical inference this is useful to simultaneously approximate the posterior and the ...
of the posterior in large samples.


Likelihood ratio and relative likelihood


Likelihood ratio

A ''likelihood ratio'' is the ratio of any two specified likelihoods, frequently written as: :\Lambda(\theta_1:\theta_2 \mid x) = \frac The likelihood ratio is central to
likelihoodist statistics Likelihoodist statistics or likelihoodism is an approach to statistics that exclusively or primarily uses the likelihood function. Likelihoodist statistics is a more minor school than the main approaches of Bayesian statistics and frequentist stat ...
: the '' law of likelihood'' states that degree to which data (considered as evidence) supports one parameter value versus another is measured by the likelihood ratio. In frequentist inference, the likelihood ratio is the basis for a test statistic, the so-called likelihood-ratio test. By the
Neyman–Pearson lemma In statistics, the Neyman–Pearson lemma was introduced by Jerzy Neyman and Egon Pearson in a paper in 1933. The Neyman-Pearson lemma is part of the Neyman-Pearson theory of statistical testing, which introduced concepts like errors of the sec ...
, this is the most powerful test for comparing two simple hypotheses at a given significance level. Numerous other tests can be viewed as likelihood-ratio tests or approximations thereof. The asymptotic distribution of the log-likelihood ratio, considered as a test statistic, is given by
Wilks' theorem In statistics Wilks' theorem offers an asymptotic distribution of the log-likelihood ratio statistic, which can be used to produce confidence intervals for maximum-likelihood estimates or as a test statistic for performing the likelihood-ratio ...
. The likelihood ratio is also of central importance in
Bayesian inference Bayesian inference is a method of statistical inference in which Bayes' theorem is used to update the probability for a hypothesis as more evidence or information becomes available. Bayesian inference is an important technique in statistics, and ...
, where it is known as the Bayes factor, and is used in
Bayes' rule In probability theory and statistics, Bayes' theorem (alternatively Bayes' law or Bayes' rule), named after Thomas Bayes, describes the probability of an event, based on prior knowledge of conditions that might be related to the event. For exa ...
. Stated in terms of
odds Odds provide a measure of the likelihood of a particular outcome. They are calculated as the ratio of the number of events that produce that outcome to the number that do not. Odds are commonly used in gambling and statistics. Odds also have ...
, Bayes' rule states that the ''posterior'' odds of two alternatives, and , given an event , is the ''prior'' odds, times the likelihood ratio. As an equation: :O(A_1:A_2 \mid B) = O(A_1:A_2) \cdot \Lambda(A_1:A_2 \mid B). The likelihood ratio is not directly used in AIC-based statistics. Instead, what is used is the relative likelihood of models (see below).


Relative likelihood function

Since the actual value of the likelihood function depends on the sample, it is often convenient to work with a standardized measure. Suppose that the maximum likelihood estimate for the parameter is \hat. Relative plausibilities of other values may be found by comparing the likelihoods of those other values with the likelihood of \hat. The ''relative likelihood'' of is defined to be (§9.3).Sprott, D. A. (2000), ''Statistical Inference in Science'', Springer (chap. 2). : R(\theta) = \frac. Thus, the relative likelihood is the likelihood ratio (discussed above) with the fixed denominator \mathcal(\hat). This corresponds to standardizing the likelihood to have a maximum of 1.


Likelihood region

A ''likelihood region'' is the set of all values of whose relative likelihood is greater than or equal to a given threshold. In terms of percentages, a ''% likelihood region'' for is defined to be. : \left\. If is a single real parameter, a % likelihood region will usually comprise an interval of real values. If the region does comprise an interval, then it is called a ''likelihood interval''.. Likelihood intervals, and more generally likelihood regions, are used for
interval estimation In statistics, interval estimation is the use of sample data to estimate an '' interval'' of plausible values of a parameter of interest. This is in contrast to point estimation, which gives a single value. The most prevalent forms of interva ...
within likelihoodist statistics: they are similar to
confidence interval In frequentist statistics, a confidence interval (CI) is a range of estimates for an unknown parameter. A confidence interval is computed at a designated ''confidence level''; the 95% confidence level is most common, but other levels, such as 9 ...
s in frequentist statistics and
credible interval In Bayesian statistics, a credible interval is an interval within which an unobserved parameter value falls with a particular probability. It is an interval in the domain of a posterior probability distribution or a predictive distribution. The ...
s in Bayesian statistics. Likelihood intervals are interpreted directly in terms of relative likelihood, not in terms of
coverage probability In statistics, the coverage probability is a technique for calculating a confidence interval which is the proportion of the time that the interval contains the true value of interest. For example, suppose our interest is in the mean number of m ...
(frequentism) or
posterior probability The posterior probability is a type of conditional probability that results from updating the prior probability with information summarized by the likelihood via an application of Bayes' rule. From an epistemological perspective, the posterior p ...
(Bayesianism). Given a model, likelihood intervals can be compared to confidence intervals. If is a single real parameter, then under certain conditions, a 14.65% likelihood interval (about 1:7 likelihood) for will be the same as a 95% confidence interval (19/20 coverage probability). In a slightly different formulation suited to the use of log-likelihoods (see
Wilks' theorem In statistics Wilks' theorem offers an asymptotic distribution of the log-likelihood ratio statistic, which can be used to produce confidence intervals for maximum-likelihood estimates or as a test statistic for performing the likelihood-ratio ...
), the test statistic is twice the difference in log-likelihoods and the probability distribution of the test statistic is approximately a
chi-squared distribution In probability theory and statistics, the chi-squared distribution (also chi-square or \chi^2-distribution) with k degrees of freedom is the distribution of a sum of the squares of k independent standard normal random variables. The chi-squar ...
with degrees-of-freedom (df) equal to the difference in df's between the two models (therefore, the −2 likelihood interval is the same as the 0.954 confidence interval; assuming difference in df's to be 1).


Likelihoods that eliminate nuisance parameters

In many cases, the likelihood is a function of more than one parameter but interest focuses on the estimation of only one, or at most a few of them, with the others being considered as
nuisance parameter Nuisance (from archaic ''nocence'', through Fr. ''noisance'', ''nuisance'', from Lat. ''nocere'', "to hurt") is a common law tort. It means that which causes offence, annoyance, trouble or injury. A nuisance can be either public (also "commo ...
s. Several alternative approaches have been developed to eliminate such nuisance parameters, so that a likelihood can be written as a function of only the parameter (or parameters) of interest: the main approaches are profile, conditional, and marginal likelihoods. These approaches are also useful when a high-dimensional likelihood surface needs to be reduced to one or two parameters of interest in order to allow a graph.


Profile likelihood

It is possible to reduce the dimensions by concentrating the likelihood function for a subset of parameters by expressing the nuisance parameters as functions of the parameters of interest and replacing them in the likelihood function. In general, for a likelihood function depending on the parameter vector \mathbf that can be partitioned into \mathbf = \left( \mathbf_ : \mathbf_ \right), and where a correspondence \mathbf_ = \mathbf_ \left( \mathbf_ \right) can be determined explicitly, concentration reduces computational burden of the original maximization problem. For instance, in a
linear regression In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables (also known as dependent and independent variables). The case of one explanatory variable is cal ...
with normally distributed errors, \mathbf = \mathbf \beta + u, the coefficient vector could be partitioned into \beta = \left \beta_ : \beta_ \right/math> (and consequently the design matrix \mathbf = \left \mathbf_ : \mathbf_ \right/math>). Maximizing with respect to \beta_ yields an optimal value function \beta_ (\beta_) = \left( \mathbf_^ \mathbf_ \right)^ \mathbf_^ \left( \mathbf - \mathbf_ \beta_ \right). Using this result, the maximum likelihood estimator for \beta_ can then be derived as :\hat_ = \left( \mathbf_^ \left( \mathbf - \mathbf_ \right) \mathbf_ \right)^ \mathbf_^ \left( \mathbf - \mathbf_ \right) \mathbf where \mathbf_ = \mathbf_ \left( \mathbf_^ \mathbf_ \right)^ \mathbf_^ is the projection matrix of \mathbf_. This result is known as the Frisch–Waugh–Lovell theorem. Since graphically the procedure of concentration is equivalent to slicing the likelihood surface along the ridge of values of the nuisance parameter \beta_ that maximizes the likelihood function, creating an isometric profile of the likelihood function for a given \beta_, the result of this procedure is also known as ''profile likelihood''. In addition to being graphed, the profile likelihood can also be used to compute
confidence interval In frequentist statistics, a confidence interval (CI) is a range of estimates for an unknown parameter. A confidence interval is computed at a designated ''confidence level''; the 95% confidence level is most common, but other levels, such as 9 ...
s that often have better small-sample properties than those based on asymptotic standard errors calculated from the full likelihood.


Conditional likelihood

Sometimes it is possible to find a
sufficient statistic In statistics, a statistic is ''sufficient'' with respect to a statistical model and its associated unknown parameter if "no other statistic that can be calculated from the same sample provides any additional information as to the value of the para ...
for the nuisance parameters, and conditioning on this statistic results in a likelihood which does not depend on the nuisance parameters. One example occurs in 2×2 tables, where conditioning on all four marginal totals leads to a conditional likelihood based on the non-central hypergeometric distribution. This form of conditioning is also the basis for
Fisher's exact test Fisher's exact test is a statistical significance test used in the analysis of contingency tables. Although in practice it is employed when sample sizes are small, it is valid for all sample sizes. It is named after its inventor, Ronald Fisher, a ...
.


Marginal likelihood

Sometimes we can remove the nuisance parameters by considering a likelihood based on only part of the information in the data, for example by using the set of ranks rather than the numerical values. Another example occurs in linear mixed models, where considering a likelihood for the residuals only after fitting the fixed effects leads to residual maximum likelihood estimation of the variance components.


Partial likelihood

A partial likelihood is an adaption of the full likelihood such that only a part of the parameters (the parameters of interest) occur in it. It is a key component of the
proportional hazards model Proportional hazards models are a class of survival models in statistics. Survival models relate the time that passes, before some event occurs, to one or more covariates that may be associated with that quantity of time. In a proportional haza ...
: using a restriction on the hazard function, the likelihood does not contain the shape of the hazard over time.


Products of likelihoods

The likelihood, given two or more
independent Independent or Independents may refer to: Arts, entertainment, and media Artist groups * Independents (artist group), a group of modernist painters based in the New Hope, Pennsylvania, area of the United States during the early 1930s * Independe ...
events, is the product of the likelihoods of each of the individual events: :\Lambda(A \mid X_1 \land X_2) = \Lambda(A \mid X_1) \cdot \Lambda(A \mid X_2) This follows from the definition of independence in probability: the probabilities of two independent events happening, given a model, is the product of the probabilities. This is particularly important when the events are from
independent and identically distributed random variables In probability theory and statistics, a collection of random variables is independent and identically distributed if each random variable has the same probability distribution as the others and all are mutually independent. This property is usu ...
, such as independent observations or sampling with replacement. In such a situation, the likelihood function factors into a product of individual likelihood functions. The empty product has value 1, which corresponds to the likelihood, given no event, being 1: before any data, the likelihood is always 1. This is similar to a uniform prior in Bayesian statistics, but in likelihoodist statistics this is not an
improper prior In Bayesian statistical inference, a prior probability distribution, often simply called the prior, of an uncertain quantity is the probability distribution that would express one's beliefs about this quantity before some evidence is taken into ...
because likelihoods are not integrated.


Log-likelihood

''Log-likelihood function'' is a logarithmic transformation of the likelihood function, often denoted by a lowercase or , to contrast with the uppercase or \mathcal for the likelihood. Because logarithms are
strictly increasing In mathematics, a monotonic function (or monotone function) is a function between ordered sets that preserves or reverses the given order. This concept first arose in calculus, and was later generalized to the more abstract setting of orde ...
functions, maximizing the likelihood is equivalent to maximizing the log-likelihood. But for practical purposes it is more convenient to work with the log-likelihood function in maximum likelihood estimation, in particular since most common
probability distribution In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomenon ...
s—notably the exponential family—are only logarithmically concave, and
concavity In calculus, the second derivative, or the second order derivative, of a function is the derivative of the derivative of . Roughly speaking, the second derivative measures how the rate of change of a quantity is itself changing; for example, ...
of the objective function plays a key role in the maximization. Given the independence of each event, the overall log-likelihood of intersection equals the sum of the log-likelihoods of the individual events. This is analogous to the fact that the overall
log-probability In probability theory and computer science, a log probability is simply a logarithm of a probability. The use of log probabilities means representing probabilities on a logarithmic scale, instead of the standard , 1/math> unit interval. Sin ...
is the sum of the log-probability of the individual events. In addition to the mathematical convenience from this, the adding process of log-likelihood has an intuitive interpretation, as often expressed as "support" from the data. When the parameters are estimated using the log-likelihood for the maximum likelihood estimation, each data point is used by being added to the total log-likelihood. As the data can be viewed as an evidence that support the estimated parameters, this process can be interpreted as "support from independent evidence ''adds",'' and the log-likelihood is the "weight of evidence". Interpreting negative log-probability as information content or surprisal, the support (log-likelihood) of a model, given an event, is the negative of the surprisal of the event, given the model: a model is supported by an event to the extent that the event is unsurprising, given the model. A logarithm of a likelihood ratio is equal to the difference of the log-likelihoods: :\log \frac = \log L(A) - \log L(B) = \ell(A) - \ell(B). Just as the likelihood, given no event, being 1, the log-likelihood, given no event, is 0, which corresponds to the value of the empty sum: without any data, there is no support for any models.


Graph

The graph of the log-likelihood is called the support curve (in the univariate case).. In the multivariate case, the concept generalizes into a support surface over the
parameter space The parameter space is the space of possible parameter values that define a particular mathematical model, often a subset of finite-dimensional Euclidean space. Often the parameters are inputs of a function, in which case the technical term for ...
. It has a relation to, but is distinct from, the support of a distribution. The term was coined by A. W. F. Edwards in the context of
statistical hypothesis testing A statistical hypothesis test is a method of statistical inference used to decide whether the data at hand sufficiently support a particular hypothesis. Hypothesis testing allows us to make probabilistic statements about population parameters. ...
, i.e. whether or not the data "support" one hypothesis (or parameter value) being tested more than any other. The log-likelihood function being plotted is used in the computation of the
score Score or scorer may refer to: *Test score, the result of an exam or test Business * Score Digital, now part of Bauer Radio * Score Entertainment, a former American trading card design and manufacturing company * Score Media, a former Canadian ...
(the gradient of the log-likelihood) and Fisher information (the curvature of the log-likelihood). This, the graph has a direct interpretation in the context of maximum likelihood estimation and likelihood-ratio tests.


Likelihood equations

If the log-likelihood function is smooth, its
gradient In vector calculus, the gradient of a scalar-valued differentiable function of several variables is the vector field (or vector-valued function) \nabla f whose value at a point p is the "direction and rate of fastest increase". If the gr ...
with respect to the parameter, known as the
score Score or scorer may refer to: *Test score, the result of an exam or test Business * Score Digital, now part of Bauer Radio * Score Entertainment, a former American trading card design and manufacturing company * Score Media, a former Canadian ...
and written s_(\theta) \equiv \nabla_ \ell_(\theta), exists and allows for the application of
differential calculus In mathematics, differential calculus is a subfield of calculus that studies the rates at which quantities change. It is one of the two traditional divisions of calculus, the other being integral calculus—the study of the area beneath a curve ...
. The basic way to maximize a differentiable function is to find the stationary points (the points where the
derivative In mathematics, the derivative of a function of a real variable measures the sensitivity to change of the function value (output value) with respect to a change in its argument (input value). Derivatives are a fundamental tool of calculus. ...
is zero); since the derivative of a sum is just the sum of the derivatives, but the derivative of a product requires the product rule, it is easier to compute the stationary points of the log-likelihood of independent events than for the likelihood of independent events. The equations defined by the stationary point of the score function serve as estimating equations for the maximum likelihood estimator. :s_(\theta) = \mathbf In that sense, the maximum likelihood estimator is implicitly defined by the value at \mathbf of the inverse function s_^: \mathbb^ \to \Theta, where \mathbb^ is the d-dimensional
Euclidean space Euclidean space is the fundamental space of geometry, intended to represent physical space. Originally, that is, in Euclid's ''Elements'', it was the three-dimensional space of Euclidean geometry, but in modern mathematics there are Euclidea ...
, and \Theta is the parameter space. Using the inverse function theorem, it can be shown that s_^ is well-defined in an open neighborhood about \mathbf with probability going to one, and \hat_ = s_^(\mathbf) is a consistent estimate of \theta. As a consequence there exists a sequence \left\ such that s_(\hat_) = \mathbf asymptotically
almost surely In probability theory, an event is said to happen almost surely (sometimes abbreviated as a.s.) if it happens with probability 1 (or Lebesgue measure 1). In other words, the set of possible exceptions may be non-empty, but it has probability 0. ...
, and \hat_ \xrightarrow \theta_. A similar result can be established using
Rolle's theorem In calculus, Rolle's theorem or Rolle's lemma essentially states that any real-valued differentiable function that attains equal values at two distinct points must have at least one stationary point somewhere between them—that is, a point whe ...
. The second derivative evaluated at \hat, known as Fisher information, determines the curvature of the likelihood surface, and thus indicates the precision of the estimate.


Exponential families

The log-likelihood is also particularly useful for exponential families of distributions, which include many of the common parametric probability distributions. The probability distribution function (and thus likelihood function) for exponential families contain products of factors involving
exponentiation Exponentiation is a mathematical operation, written as , involving two numbers, the '' base'' and the ''exponent'' or ''power'' , and pronounced as " (raised) to the (power of) ". When is a positive integer, exponentiation corresponds to ...
. The logarithm of such a function is a sum of products, again easier to differentiate than the original function. An exponential family is one whose probability density function is of the form (for some functions, writing \langle -, - \rangle for the
inner product In mathematics, an inner product space (or, rarely, a Hausdorff pre-Hilbert space) is a real vector space or a complex vector space with an operation called an inner product. The inner product of two vectors in the space is a scalar, often ...
): : p(x \mid \boldsymbol \theta) = h(x) \exp\Big(\langle \boldsymbol\eta(), \mathbf(x)\rangle -A() \Big). Each of these terms has an interpretation, but simply switching from probability to likelihood and taking logarithms yields the sum: : \ell(\boldsymbol \theta \mid x) = \langle \boldsymbol\eta(), \mathbf(x)\rangle - A() + \log h(x). The \boldsymbol \eta(\boldsymbol \theta) and h(x) each correspond to a
change of coordinates In mathematics, an ordered basis of a vector space of finite dimension allows representing uniquely any element of the vector space by a coordinate vector, which is a sequence of scalars called coordinates. If two different bases are consi ...
, so in these coordinates, the log-likelihood of an exponential family is given by the simple formula: : \ell(\boldsymbol \eta \mid x) = \langle \boldsymbol\eta, \mathbf(x)\rangle - A(). In words, the log-likelihood of an exponential family is inner product of the natural parameter and the
sufficient statistic In statistics, a statistic is ''sufficient'' with respect to a statistical model and its associated unknown parameter if "no other statistic that can be calculated from the same sample provides any additional information as to the value of the para ...
, minus the normalization factor ( log-partition function) . Thus for example the maximum likelihood estimate can be computed by taking derivatives of the sufficient statistic and the log-partition function .


Example: the gamma distribution

The
gamma distribution In probability theory and statistics, the gamma distribution is a two- parameter family of continuous probability distributions. The exponential distribution, Erlang distribution, and chi-square distribution are special cases of the gamma di ...
is an exponential family with two parameters, \alpha and \beta. The likelihood function is :\mathcal (\alpha, \beta \mid x) = \frac x^ e^. Finding the maximum likelihood estimate of \beta for a single observed value x looks rather daunting. Its logarithm is much simpler to work with: :\log \mathcal(\alpha,\beta \mid x) = \alpha \log \beta - \log \Gamma(\alpha) + (\alpha-1) \log x - \beta x. \, To maximize the log-likelihood, we first take the
partial derivative In mathematics, a partial derivative of a function of several variables is its derivative with respect to one of those variables, with the others held constant (as opposed to the total derivative, in which all variables are allowed to vary). Pa ...
with respect to \beta: :\frac = \frac - x. If there are a number of independent observations x_1, \ldots, x_n, then the joint log-likelihood will be the sum of individual log-likelihoods, and the derivative of this sum will be a sum of derivatives of each individual log-likelihood: : \begin & \frac \\ = & \frac + \cdots + \frac = \frac \beta - \sum_^n x_i. \end To complete the maximization procedure for the joint log-likelihood, the equation is set to zero and solved for \beta: :\widehat\beta = \frac. Here \widehat\beta denotes the maximum-likelihood estimate, and \textstyle \bar = \frac \sum_^n x_i is the sample mean of the observations.


Background and interpretation


Historical remarks

The term "likelihood" has been in use in English since at least late
Middle English Middle English (abbreviated to ME) is a form of the English language that was spoken after the Norman conquest of 1066, until the late 15th century. The English language underwent distinct variations and developments following the Old Englis ...
. Its formal use to refer to a specific function in mathematical statistics was proposed by Ronald Fisher, in two research papers published in 1921 and 1922. The 1921 paper introduced what is today called a "likelihood interval"; the 1922 paper introduced the term "
method of maximum likelihood In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statis ...
". Quoting Fisher: The concept of likelihood should not be confused with probability as mentioned by Sir Ronald Fisher Fisher's invention of statistical likelihood was in reaction against an earlier form of reasoning called inverse probability. His use of the term "likelihood" fixed the meaning of the term within mathematical statistics. A. W. F. Edwards (1972) established the axiomatic basis for use of the log-likelihood ratio as a measure of relative support for one hypothesis against another. The ''support function'' is then the natural logarithm of the likelihood function. Both terms are used in
phylogenetics In biology, phylogenetics (; from Greek φυλή/ φῦλον [] "tribe, clan, race", and wikt:γενετικός, γενετικός [] "origin, source, birth") is the study of the evolutionary history and relationships among or within groups ...
, but were not adopted in a general treatment of the topic of statistical evidence.


Interpretations under different foundations

Among statisticians, there is no consensus about what the foundation of statistics should be. There are four main paradigms that have been proposed for the foundation: frequentism, Bayesianism, likelihoodism, and AIC-based. For each of the proposed foundations, the interpretation of likelihood is different. The four interpretations are described in the subsections below.


Frequentist interpretation


Bayesian interpretation

In
Bayesian inference Bayesian inference is a method of statistical inference in which Bayes' theorem is used to update the probability for a hypothesis as more evidence or information becomes available. Bayesian inference is an important technique in statistics, and ...
, although one can speak about the likelihood of any proposition or
random variable A random variable (also called random quantity, aleatory variable, or stochastic variable) is a mathematical formalization of a quantity or object which depends on random events. It is a mapping or a function from possible outcomes (e.g., the po ...
given another random variable: for example the likelihood of a parameter value or of a
statistical model A statistical model is a mathematical model that embodies a set of statistical assumptions concerning the generation of sample data (and similar data from a larger population). A statistical model represents, often in considerably idealized form ...
(see marginal likelihood), given specified data or other evidence,I. J. Good: ''Probability and the Weighing of Evidence'' (Griffin 1950), §6.1H. Jeffreys: ''Theory of Probability'' (3rd ed., Oxford University Press 1983), §1.22E. T. Jaynes: ''Probability Theory: The Logic of Science'' (Cambridge University Press 2003), §4.1D. V. Lindley: ''Introduction to Probability and Statistics from a Bayesian Viewpoint. Part 1: Probability'' (Cambridge University Press 1980), §1.6 the likelihood function remains the same entity, with the additional interpretations of (i) a
conditional density In probability theory and statistics, given two jointly distributed random variables X and Y, the conditional probability distribution of Y given X is the probability distribution of Y when X is known to be a particular value; in some cases the c ...
of the data given the parameter (since the parameter is then a random variable) and (ii) a measure or amount of information brought by the data about the parameter value or even the model.A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, D. B. Rubin: ''Bayesian Data Analysis'' (3rd ed., Chapman & Hall/CRC 2014), §1.3 Due to the introduction of a probability structure on the parameter space or on the collection of models, it is possible that a parameter value or a statistical model have a large likelihood value for given data, and yet have a low ''probability'', or vice versa. This is often the case in medical contexts. Following
Bayes' Rule In probability theory and statistics, Bayes' theorem (alternatively Bayes' law or Bayes' rule), named after Thomas Bayes, describes the probability of an event, based on prior knowledge of conditions that might be related to the event. For exa ...
, the likelihood when seen as a conditional density can be multiplied by the prior probability density of the parameter and then normalized, to give a
posterior probability The posterior probability is a type of conditional probability that results from updating the prior probability with information summarized by the likelihood via an application of Bayes' rule. From an epistemological perspective, the posterior p ...
density. More generally, the likelihood of an unknown quantity X given another unknown quantity Y is proportional to the ''probability of Y given X''.


Likelihoodist interpretation

In frequentist statistics, the likelihood function is itself a
statistic A statistic (singular) or sample statistic is any quantity computed from values in a sample which is considered for a statistical purpose. Statistical purposes include estimating a population parameter, describing a sample, or evaluating a hypo ...
that summarizes a single sample from a population, whose calculated value depends on a choice of several parameters ''θ''1 ... ''θ''p, where ''p'' is the count of parameters in some already-selected
statistical model A statistical model is a mathematical model that embodies a set of statistical assumptions concerning the generation of sample data (and similar data from a larger population). A statistical model represents, often in considerably idealized form ...
. The value of the likelihood serves as a figure of merit for the choice used for the parameters, and the parameter set with maximum likelihood is the best choice, given the data available. The specific calculation of the likelihood is the probability that the observed sample would be assigned, assuming that the model chosen and the values of the several parameters ''θ'' give an accurate approximation of the frequency distribution of the population that the observed sample was drawn from. Heuristically, it makes sense that a good choice of parameters is those which render the sample actually observed the maximum possible ''post-hoc'' probability of having happened.
Wilks' theorem In statistics Wilks' theorem offers an asymptotic distribution of the log-likelihood ratio statistic, which can be used to produce confidence intervals for maximum-likelihood estimates or as a test statistic for performing the likelihood-ratio ...
quantifies the heuristic rule by showing that the difference in the logarithm of the likelihood generated by the estimate's parameter values and the logarithm of the likelihood generated by population's "true" (but unknown) parameter values is asymptotically χ2 distributed. Each independent sample's maximum likelihood estimate is a separate estimate of the "true" parameter set describing the population sampled. Successive estimates from many independent samples will cluster together with the population's "true" set of parameter values hidden somewhere in their midst. The difference in the logarithms of the maximum likelihood and adjacent parameter sets' likelihoods may be used to draw a
confidence region In statistics, a confidence region is a multi-dimensional generalization of a confidence interval. It is a set of points in an ''n''-dimensional space, often represented as an ellipsoid around a point which is an estimated solution to a problem, a ...
on a plot whose co-ordinates are the parameters ''θ''1 ... ''θ''p. The region surrounds the maximum-likelihood estimate, and all points (parameter sets) within that region differ at most in log-likelihood by some fixed value. The χ2 distribution given by
Wilks' theorem In statistics Wilks' theorem offers an asymptotic distribution of the log-likelihood ratio statistic, which can be used to produce confidence intervals for maximum-likelihood estimates or as a test statistic for performing the likelihood-ratio ...
converts the region's log-likelihood differences into the "confidence" that the population's "true" parameter set lies inside. The art of choosing the fixed log-likelihood difference is to make the confidence acceptably high while keeping the region acceptably small (narrow range of estimates). As more data are observed, instead of being used to make independent estimates, they can be combined with the previous samples to make a single combined sample, and that large sample may be used for a new maximum likelihood estimate. As the size of the combined sample increases, the size of the likelihood region with the same confidence shrinks. Eventually, either the size of the confidence region is very nearly a single point, or the entire population has been sampled; in both cases, the estimated parameter set is essentially the same as the population parameter set.


AIC-based interpretation

Under the AIC paradigm, likelihood is interpreted within the context of
information theory Information theory is the scientific study of the quantification, storage, and communication of information. The field was originally established by the works of Harry Nyquist and Ralph Hartley, in the 1920s, and Claude Shannon in the 1940s. ...
.


See also


Notes


References


Further reading

* * * * * * * *


External links


Likelihood function at Planetmath
* {{DEFAULTSORT:Likelihood Function Bayesian statistics