Maximum Likelihood
   HOME

TheInfoList



OR:

In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed
probability distribution In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomeno ...
, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statistical model, the observed data is most probable. The point in the parameter space that maximizes the likelihood function is called the maximum likelihood estimate. The logic of maximum likelihood is both intuitive and flexible, and as such the method has become a dominant means of statistical inference. If the likelihood function is differentiable, the derivative test for finding maxima can be applied. In some cases, the first-order conditions of the likelihood function can be solved analytically; for instance, the ordinary least squares estimator for a
linear regression In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables (also known as dependent and independent variables). The case of one explanatory variable is ...
model maximizes the likelihood when all observed outcomes are assumed to have Normal distributions with the same variance. From the perspective of Bayesian inference, MLE is generally equivalent to maximum a posteriori (MAP) estimation with uniform prior distributions (or a normal prior distribution with a standard deviation of infinity). In
frequentist inference Frequentist inference is a type of statistical inference based in frequentist probability, which treats “probability” in equivalent terms to “frequency” and draws conclusions from sample-data by means of emphasizing the frequency or pro ...
, MLE is a special case of an extremum estimator, with the objective function being the likelihood.


Principles

We model a set of observations as a random sample from an unknown joint
probability distribution In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomeno ...
which is expressed in terms of a set of parameters. The goal of maximum likelihood estimation is to determine the parameters for which the observed data have the highest joint probability. We write the parameters governing the joint distribution as a vector \; \theta = \left \theta_,\, \theta_2,\, \ldots,\, \theta_k \right \; so that this distribution falls within a
parametric family In mathematics and its applications, a parametric family or a parameterized family is a family of objects (a set of related objects) whose differences depend only on the chosen values for a set of parameters. Common examples are parametrized (fa ...
\; \ \;, where \, \Theta \, is called the '' parameter space'', a finite-dimensional subset of
Euclidean space Euclidean space is the fundamental space of geometry, intended to represent physical space. Originally, that is, in Euclid's ''Elements'', it was the three-dimensional space of Euclidean geometry, but in modern mathematics there are Euclidean sp ...
. Evaluating the joint density at the observed data sample \; \mathbf = (y_1, y_2, \ldots, y_n) \; gives a real-valued function, :\mathcal_(\theta) = \mathcal_(\theta; \mathbf) = f_(\mathbf; \theta) \;, which is called the likelihood function. For
independent and identically distributed random variables In probability theory and statistics, a collection of random variables is independent and identically distributed if each random variable has the same probability distribution as the others and all are mutually independent. This property is usu ...
, f_(\mathbf; \theta) will be the product of univariate density functions: :f_(\mathbf; \theta) = \prod_^n \, f_k^\mathsf(y_k; \theta) ~. The goal of maximum likelihood estimation is to find the values of the model parameters that maximize the likelihood function over the parameter space, that is :\hat = \underset\,\mathcal_(\theta\,;\mathbf) ~. Intuitively, this selects the parameter values that make the observed data most probable. The specific value ~ \hat = \hat_(\mathbf) \in \Theta ~ that maximizes the likelihood function \, \mathcal_ \, is called the maximum likelihood estimate. Further, if the function \; \hat_ : \mathbb^ \to \Theta \; so defined is measurable, then it is called the maximum likelihood estimator. It is generally a function defined over the sample space, i.e. taking a given sample as its argument. A sufficient but not necessary condition for its existence is for the likelihood function to be continuous over a parameter space \, \Theta \, that is compact. For an open \, \Theta \, the likelihood function may increase without ever reaching a supremum value. In practice, it is often convenient to work with the
natural logarithm The natural logarithm of a number is its logarithm to the base of the mathematical constant , which is an irrational and transcendental number approximately equal to . The natural logarithm of is generally written as , , or sometimes, if ...
of the likelihood function, called the log-likelihood: : \ell(\theta\,;\mathbf) = \ln \mathcal_(\theta\,;\mathbf) ~. Since the logarithm is a
monotonic function In mathematics, a monotonic function (or monotone function) is a function between ordered sets that preserves or reverses the given order. This concept first arose in calculus, and was later generalized to the more abstract setting of ord ...
, the maximum of \; \ell(\theta\,;\mathbf) \; occurs at the same value of \theta as does the maximum of \, \mathcal_ ~. If \ell(\theta\,;\mathbf) is differentiable in \, \Theta \,, the necessary conditions for the occurrence of a maximum (or a minimum) are :\frac = 0, \quad \frac = 0, \quad \ldots, \quad \frac = 0 ~, known as the likelihood equations. For some models, these equations can be explicitly solved for \, \widehat \,, but in general no closed-form solution to the maximization problem is known or available, and an MLE can only be found via numerical optimization. Another problem is that in finite samples, there may exist multiple roots for the likelihood equations. Whether the identified root \, \widehat \, of the likelihood equations is indeed a (local) maximum depends on whether the matrix of second-order partial and cross-partial derivatives, the so-called Hessian matrix :\mathbf\left(\widehat\right) = \begin \left. \frac \_ & \left. \frac \_ & \dots & \left. \frac \_ \\ \left. \frac \_ & \left. \frac \_ & \dots & \left. \frac \_ \\ \vdots & \vdots & \ddots & \vdots \\ \left. \frac \_ & \left. \frac \_ & \dots & \left. \frac \_ \end ~, is
negative semi-definite Negative may refer to: Science and mathematics * Negative number * Negative mass * Negative energy * Negative charge, one of the two types of electric charge * Negative (electrical polarity), in electric circuits * Negative result (disambigu ...
at \widehat, as this indicates local concavity. Conveniently, most common
probability distribution In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomeno ...
s – in particular the exponential family – are logarithmically concave.


Restricted parameter space

While the domain of the likelihood function—the parameter space—is generally a finite-dimensional subset of
Euclidean space Euclidean space is the fundamental space of geometry, intended to represent physical space. Originally, that is, in Euclid's ''Elements'', it was the three-dimensional space of Euclidean geometry, but in modern mathematics there are Euclidean sp ...
, additional
restriction Restriction, restrict or restrictor may refer to: Science and technology * restrict, a keyword in the C programming language used in pointer declarations * Restriction enzyme, a type of enzyme that cleaves genetic material Mathematics and log ...
s sometimes need to be incorporated into the estimation process. The parameter space can be expressed as :\Theta = \left\ ~, where \; h(\theta) = \left h_(\theta), h_(\theta), \ldots, h_(\theta) \right\; is a vector-valued function mapping \, \mathbb^ \, into \; \mathbb^ ~. Estimating the true parameter \theta belonging to \Theta then, as a practical matter, means to find the maximum of the likelihood function subject to the
constraint Constraint may refer to: * Constraint (computer-aided design), a demarcation of geometrical characteristics between two or more entities or solid modeling bodies * Constraint (mathematics), a condition of an optimization problem that the solution ...
~h(\theta) = 0 ~. Theoretically, the most natural approach to this constrained optimization problem is the method of substitution, that is "filling out" the restrictions \; h_, h_, \ldots, h_ \; to a set \; h_, h_, \ldots, h_, h_, \ldots, h_ \; in such a way that \; h^ = \left h_, h_, \ldots, h_ \right\; is a
one-to-one function In mathematics, an injective function (also known as injection, or one-to-one function) is a function that maps distinct elements of its domain to distinct elements; that is, implies . (Equivalently, implies in the equivalent contraposi ...
from \mathbb^ to itself, and reparameterize the likelihood function by setting \; \phi_ = h_(\theta_, \theta_, \ldots, \theta_) ~. Because of the equivariance of the maximum likelihood estimator, the properties of the MLE apply to the restricted estimates also. For instance, in a
multivariate normal distribution In probability theory and statistics, the multivariate normal distribution, multivariate Gaussian distribution, or joint normal distribution is a generalization of the one-dimensional ( univariate) normal distribution to higher dimensions. One ...
the covariance matrix \, \Sigma \, must be positive-definite; this restriction can be imposed by replacing \; \Sigma = \Gamma^ \Gamma \;, where \Gamma is a real upper triangular matrix and \Gamma^ is its
transpose In linear algebra, the transpose of a matrix is an operator which flips a matrix over its diagonal; that is, it switches the row and column indices of the matrix by producing another matrix, often denoted by (among other notations). The tr ...
. In practice, restrictions are usually imposed using the method of Lagrange which, given the constraints as defined above, leads to the ''restricted likelihood equations'' :\frac - \frac \lambda = 0 and h(\theta) = 0 \;, where ~ \lambda = \left \lambda_, \lambda_, \ldots, \lambda_\right\mathsf ~ is a column-vector of Lagrange multipliers and \; \frac \; is the Jacobian matrix of partial derivatives. Naturally, if the constraints are not binding at the maximum, the Lagrange multipliers should be zero. This in turn allows for a statistical test of the "validity" of the constraint, known as the Lagrange multiplier test.


Properties

A maximum likelihood estimator is an extremum estimator obtained by maximizing, as a function of ''θ'', the
objective function In mathematical optimization and decision theory, a loss function or cost function (sometimes also called an error function) is a function that maps an event or values of one or more variables onto a real number intuitively representing some "cos ...
\widehat(\theta\,;x). If the data are independent and identically distributed, then we have : \widehat(\theta\,;x)=\frac1n \sum_^n \ln f(x_i\mid\theta), this being the sample analogue of the expected log-likelihood \ell(\theta) = \operatorname , \ln f(x_i\mid\theta) \,/math>, where this expectation is taken with respect to the true density. Maximum-likelihood estimators have no optimum properties for finite samples, in the sense that (when evaluated on finite samples) other estimators may have greater concentration around the true parameter-value. However, like other estimation methods, maximum likelihood estimation possesses a number of attractive limiting properties: As the sample size increases to infinity, sequences of maximum likelihood estimators have these properties: * Consistency: the sequence of MLEs converges in probability to the value being estimated. * Functional equivariance: If \hat is the maximum likelihood estimator for \theta , and if g(\theta) is any transformation of \theta , then the maximum likelihood estimator for \alpha = g(\theta ) is \hat = g(\hat ) . * Efficiency, i.e. it achieves the Cramér–Rao lower bound when the sample size tends to infinity. This means that no consistent estimator has lower asymptotic
mean squared error In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference betwe ...
than the MLE (or other estimators attaining this bound), which also means that MLE has asymptotic normality. * Second-order efficiency after correction for bias.


Consistency

Under the conditions outlined below, the maximum likelihood estimator is consistent. The consistency means that if the data were generated by f(\cdot\,;\theta_0) and we have a sufficiently large number of observations ''n'', then it is possible to find the value of ''θ''0 with arbitrary precision. In mathematical terms this means that as ''n'' goes to infinity the estimator \widehat converges in probability to its true value: : \widehat_\mathrm\ \xrightarrow\ \theta_0. Under slightly stronger conditions, the estimator converges
almost surely In probability theory, an event is said to happen almost surely (sometimes abbreviated as a.s.) if it happens with probability 1 (or Lebesgue measure 1). In other words, the set of possible exceptions may be non-empty, but it has probability 0 ...
(or ''strongly''): : \widehat_\mathrm\ \xrightarrow\ \theta_0. In practical applications, data is never generated by f(\cdot\,;\theta_0). Rather, f(\cdot\,;\theta_0) is a model, often in idealized form, of the process generated by the data. It is a common aphorism in statistics that '' all models are wrong''. Thus, true consistency does not occur in practical applications. Nevertheless, consistency is often considered to be a desirable property for an estimator to have. To establish consistency, the following conditions are sufficient. The dominance condition can be employed in the case of i.i.d. observations. In the non-i.i.d. case, the uniform convergence in probability can be checked by showing that the sequence \widehat(\theta\mid x) is stochastically equicontinuous. If one wants to demonstrate that the ML estimator \widehat converges to ''θ''0
almost surely In probability theory, an event is said to happen almost surely (sometimes abbreviated as a.s.) if it happens with probability 1 (or Lebesgue measure 1). In other words, the set of possible exceptions may be non-empty, but it has probability 0 ...
, then a stronger condition of uniform convergence almost surely has to be imposed: : \sup_ \left\, \;\widehat(\theta\mid x) - \ell(\theta)\;\right\, \ \xrightarrow\ 0. Additionally, if (as assumed above) the data were generated by f(\cdot\,;\theta_0), then under certain conditions, it can also be shown that the maximum likelihood estimator converges in distribution to a normal distribution. Specifically,By Theorem 3.3 in : \sqrt\left(\widehat_\mathrm - \theta_0\right)\ \xrightarrow\ \mathcal\left(0,\, I^\right) where is the Fisher information matrix.


Functional equivariance

The maximum likelihood estimator selects the parameter value which gives the observed data the largest possible probability (or probability density, in the continuous case). If the parameter consists of a number of components, then we define their separate maximum likelihood estimators, as the corresponding component of the MLE of the complete parameter. Consistent with this, if \widehat is the MLE for \theta, and if g(\theta) is any transformation of \theta, then the MLE for \alpha=g(\theta) is by definition :\widehat = g(\,\widehat\,). \, It maximizes the so-called profile likelihood: :\bar(\alpha) = \sup_ L(\theta). \, The MLE is also equivariant with respect to certain transformations of the data. If y=g(x) where g is one to one and does not depend on the parameters to be estimated, then the density functions satisfy :f_Y(y) = \frac and hence the likelihood functions for X and Y differ only by a factor that does not depend on the model parameters. For example, the MLE parameters of the log-normal distribution are the same as those of the normal distribution fitted to the logarithm of the data.


Efficiency

As assumed above, if the data were generated by ~f(\cdot\,;\theta_0)~, then under certain conditions, it can also be shown that the maximum likelihood estimator converges in distribution to a normal distribution. It is -consistent and asymptotically efficient, meaning that it reaches the Cramér–Rao bound. Specifically, : \sqrt \, \left( \widehat_\text - \theta_0 \right)\ \ \xrightarrow\ \ \mathcal \left( 0,\ \mathcal^ \right) ~, where ~\mathcal~ is the Fisher information matrix: : \mathcal_ = \operatorname \, \biggl \; - \; \biggr~. In particular, it means that the
bias Bias is a disproportionate weight ''in favor of'' or ''against'' an idea or thing, usually in a way that is closed-minded, prejudicial, or unfair. Biases can be innate or learned. People may develop biases for or against an individual, a group ...
of the maximum likelihood estimator is equal to zero up to the order .


Second-order efficiency after correction for bias

However, when we consider the higher-order terms in the expansion of the distribution of this estimator, it turns out that has bias of order . This bias is equal to (componentwise) : b_h \; \equiv \; \operatorname \biggl \; \left( \widehat\theta_\mathrm - \theta_0 \right)_h \; \biggr \; = \; \frac \, \sum_^m \; \mathcal^ \; \mathcal^ \left( \frac \, K_ \; + \; J_ \right) where \mathcal^ (with superscripts) denotes the (''j,k'')-th component of the ''inverse'' Fisher information matrix \mathcal^, and : \frac \, K_ \; + \; J_ \; = \; \operatorname\,\biggl ; \frac12 \frac + \frac\,\frac \; \biggr~ . Using these formulae it is possible to estimate the second-order bias of the maximum likelihood estimator, and ''correct'' for that bias by subtracting it: : \widehat^*_\text = \widehat_\text - \widehat ~ . This estimator is unbiased up to the terms of order , and is called the bias-corrected maximum likelihood estimator. This bias-corrected estimator is (at least within the curved exponential family), meaning that it has minimal mean squared error among all second-order bias-corrected estimators, up to the terms of the order  . It is possible to continue this process, that is to derive the third-order bias-correction term, and so on. However, the maximum likelihood estimator is ''not'' third-order efficient.


Relation to Bayesian inference

A maximum likelihood estimator coincides with the most probable Bayesian estimator given a uniform prior distribution on the parameters. Indeed, the maximum a posteriori estimate is the parameter that maximizes the probability of given the data, given by Bayes' theorem: : \operatorname(\theta\mid x_1,x_2,\ldots,x_n) = \frac where \operatorname(\theta) is the prior distribution for the parameter and where \operatorname(x_1,x_2,\ldots,x_n) is the probability of the data averaged over all parameters. Since the denominator is independent of , the Bayesian estimator is obtained by maximizing f(x_1,x_2,\ldots,x_n\mid\theta)\operatorname(\theta) with respect to . If we further assume that the prior \operatorname(\theta) is a uniform distribution, the Bayesian estimator is obtained by maximizing the likelihood function f(x_1,x_2,\ldots,x_n\mid\theta). Thus the Bayesian estimator coincides with the maximum likelihood estimator for a uniform prior distribution \operatorname(\theta).


Application of maximum-likelihood estimation in Bayes decision theory

In many practical applications in
machine learning Machine learning (ML) is a field of inquiry devoted to understanding and building methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. It is seen as a part of artificial intelligence. Machine ...
, maximum-likelihood estimation is used as the model for parameter estimation. The Bayesian Decision theory is about designing a classifier that minimizes total expected risk, especially, when the costs (the loss function) associated with different decisions are equal, the classifier is minimizing the error over the whole distribution. Thus, the Bayes Decision Rule is stated as :"decide \;w_1\; if ~\operatorname(w_1, x) \; > \; \operatorname(w_2, x)~;~ otherwise decide \;w_2\;" where \;w_1\,, w_2\; are predictions of different classes. From a perspective of minimizing error, it can also be stated as :w = \underset \; \int_^\infty \operatorname(\text\mid x)\operatorname(x)\,\operatornamex~ where :\operatorname(\text\mid x) = \operatorname(w_1\mid x)~ if we decide \;w_2\; and \;\operatorname(\text\mid x) = \operatorname(w_2\mid x)\; if we decide \;w_1\;. By applying Bayes' theorem :\operatorname(w_i \mid x) = \frac, and if we further assume the zero-or-one loss function, which is a same loss for all errors, the Bayes Decision rule can be reformulated as: :h_\text = \underset \, \bigl , \operatorname(x\mid w)\,\operatorname(w) \,\bigr;, where h_\text is the prediction and \;\operatorname(w)\; is the
prior probability In Bayesian statistical inference, a prior probability distribution, often simply called the prior, of an uncertain quantity is the probability distribution that would express one's beliefs about this quantity before some evidence is taken into ...
.


Relation to minimizing Kullback–Leibler divergence and cross entropy

Finding \hat \theta that maximizes the likelihood is asymptotically equivalent to finding the \hat \theta that defines a probability distribution (Q_) that has a minimal distance, in terms of
Kullback–Leibler divergence In mathematical statistics, the Kullback–Leibler divergence (also called relative entropy and I-divergence), denoted D_\text(P \parallel Q), is a type of statistical distance: a measure of how one probability distribution ''P'' is different fro ...
, to the real probability distribution from which our data were generated (i.e., generated by P_). In an ideal world, P and Q are the same (and the only thing unknown is \theta that defines P), but even if they are not and the model we use is misspecified, still the MLE will give us the "closest" distribution (within the restriction of a model Q that depends on \hat \theta) to the real distribution P_. Since cross entropy is just
Shannon's entropy In information theory, the entropy of a random variable is the average level of "information", "surprise", or "uncertainty" inherent to the variable's possible outcomes. Given a discrete random variable X, which takes values in the alphabet \ ...
plus KL divergence, and since the entropy of P_ is constant, then the MLE is also asymptotically minimizing cross entropy.


Examples


Discrete uniform distribution

Consider a case where ''n'' tickets numbered from 1 to ''n'' are placed in a box and one is selected at random (''see
uniform distribution Uniform distribution may refer to: * Continuous uniform distribution * Discrete uniform distribution * Uniform distribution (ecology) * Equidistributed sequence In mathematics, a sequence (''s''1, ''s''2, ''s''3, ...) of real numbers is said to be ...
''); thus, the sample size is 1. If ''n'' is unknown, then the maximum likelihood estimator \widehat of ''n'' is the number ''m'' on the drawn ticket. (The likelihood is 0 for ''n'' < ''m'', for ''n'' ≥ ''m'', and this is greatest when ''n'' = ''m''. Note that the maximum likelihood estimate of ''n'' occurs at the lower extreme of possible values , rather than somewhere in the "middle" of the range of possible values, which would result in less bias.) The
expected value In probability theory, the expected value (also called expectation, expectancy, mathematical expectation, mean, average, or first moment) is a generalization of the weighted average. Informally, the expected value is the arithmetic mean of a ...
of the number ''m'' on the drawn ticket, and therefore the expected value of \widehat, is (''n'' + 1)/2. As a result, with a sample size of 1, the maximum likelihood estimator for ''n'' will systematically underestimate ''n'' by (''n'' − 1)/2.


Discrete distribution, finite parameter space

Suppose one wishes to determine just how biased an
unfair coin In probability theory and statistics, a sequence of independent Bernoulli trials with probability 1/2 of success on each trial is metaphorically called a fair coin. One for which the probability is not 1/2 is called a biased or unfair coin. In the ...
is. Call the probability of tossing a ‘ head’ ''p''. The goal then becomes to determine ''p''. Suppose the coin is tossed 80 times: i.e. the sample might be something like ''x''1 = H, ''x''2 = T, ..., ''x''80 = T, and the count of the number of heads "H" is observed. The probability of tossing tails is 1 − ''p'' (so here ''p'' is ''θ'' above). Suppose the outcome is 49 heads and 31  tails, and suppose the coin was taken from a box containing three coins: one which gives heads with probability ''p'' = , one which gives heads with probability ''p'' =  and another which gives heads with probability ''p'' = . The coins have lost their labels, so which one it was is unknown. Using maximum likelihood estimation, the coin that has the largest likelihood can be found, given the data that were observed. By using the probability mass function of the
binomial distribution In probability theory and statistics, the binomial distribution with parameters ''n'' and ''p'' is the discrete probability distribution of the number of successes in a sequence of ''n'' independent experiments, each asking a yes–no qu ...
with sample size equal to 80, number successes equal to 49 but for different values of ''p'' (the "probability of success"), the likelihood function (defined below) takes one of three values: : \begin \operatorname\bigl ;\mathrm = 49 \mid p=\tfrac\;\bigr& = \binom(\tfrac)^(1-\tfrac)^ \approx 0.000, \\ pt\operatorname\bigl ;\mathrm = 49 \mid p=\tfrac\;\bigr& = \binom(\tfrac)^(1-\tfrac)^ \approx 0.012, \\ pt\operatorname\bigl ;\mathrm = 49 \mid p=\tfrac\;\bigr& = \binom(\tfrac)^(1-\tfrac)^ \approx 0.054~. \end The likelihood is maximized when  = , and so this is the ''maximum likelihood estimate'' for .


Discrete distribution, continuous parameter space

Now suppose that there was only one coin but its could have been any value The likelihood function to be maximised is : L(p) = f_D(\mathrm = 49 \mid p) = \binom p^(1 - p)^~, and the maximisation is over all possible values One way to maximize this function is by differentiating with respect to and setting to zero: : \begin 0 & = \frac \left( \binom p^(1-p)^ \right)~, \\ pt0 & = 49 p^(1-p)^ - 31 p^(1-p)^ \\ pt & = p^(1-p)^\left 49 (1-p) - 31 p \right \\ pt & = p^(1-p)^\left 49 - 80 p \right. \end This is a product of three terms. The first term is 0 when  = 0. The second is 0 when  = 1. The third is zero when  = . The solution that maximizes the likelihood is clearly  =  (since  = 0 and  = 1 result in a likelihood of 0). Thus the ''maximum likelihood estimator'' for is . This result is easily generalized by substituting a letter such as in the place of 49 to represent the observed number of 'successes' of our
Bernoulli trial In the theory of probability and statistics, a Bernoulli trial (or binomial trial) is a random experiment with exactly two possible outcomes, "success" and "failure", in which the probability of success is the same every time the experiment is ...
s, and a letter such as in the place of 80 to represent the number of Bernoulli trials. Exactly the same calculation yields which is the maximum likelihood estimator for any sequence of Bernoulli trials resulting in 'successes'.


Continuous distribution, continuous parameter space

For the
normal distribution In statistics, a normal distribution or Gaussian distribution is a type of continuous probability distribution for a real-valued random variable. The general form of its probability density function is : f(x) = \frac e^ The parameter \mu i ...
\mathcal(\mu, \sigma^2) which has
probability density function In probability theory, a probability density function (PDF), or density of a continuous random variable, is a function whose value at any given sample (or point) in the sample space (the set of possible values taken by the random variable) c ...
:f(x\mid \mu,\sigma^2) = \frac \exp\left(-\frac \right), the corresponding
probability density function In probability theory, a probability density function (PDF), or density of a continuous random variable, is a function whose value at any given sample (or point) in the sample space (the set of possible values taken by the random variable) c ...
for a sample of independent identically distributed normal random variables (the likelihood) is :f(x_1,\ldots,x_n \mid \mu,\sigma^2) = \prod_^n f( x_i\mid \mu, \sigma^2) = \left( \frac \right)^ \exp\left( -\frac\right). This family of distributions has two parameters: ; so we maximize the likelihood, \mathcal (\mu,\sigma^2) = f(x_1,\ldots,x_n \mid \mu, \sigma^2), over both parameters simultaneously, or if possible, individually. Since the
logarithm In mathematics, the logarithm is the inverse function to exponentiation. That means the logarithm of a number  to the base  is the exponent to which must be raised, to produce . For example, since , the ''logarithm base'' 10 of ...
function itself is a continuous
strictly increasing In mathematics, a monotonic function (or monotone function) is a function between ordered sets that preserves or reverses the given order. This concept first arose in calculus, and was later generalized to the more abstract setting of order ...
function over the range of the likelihood, the values which maximize the likelihood will also maximize its logarithm (the log-likelihood itself is not necessarily strictly increasing). The log-likelihood can be written as follows: : \log\Bigl( \mathcal (\mu,\sigma^2)\Bigr) = -\frac \log(2\pi\sigma^2) - \frac \sum_^n (\,x_i-\mu\,)^2 (Note: the log-likelihood is closely related to information entropy and Fisher information.) We now compute the derivatives of this log-likelihood as follows. : \begin 0 & = \frac \log\Bigl( \mathcal (\mu,\sigma^2)\Bigr) = 0 - \frac. \end where \bar is the
sample mean The sample mean (or "empirical mean") and the sample covariance are statistics computed from a sample of data on one or more random variables. The sample mean is the average value (or mean value) of a sample of numbers taken from a larger po ...
. This is solved by :\widehat\mu = \bar = \sum^n_ \frac. This is indeed the maximum of the function, since it is the only turning point in and the second derivative is strictly less than zero. Its
expected value In probability theory, the expected value (also called expectation, expectancy, mathematical expectation, mean, average, or first moment) is a generalization of the weighted average. Informally, the expected value is the arithmetic mean of a ...
is equal to the parameter of the given distribution, :\operatorname\bigl ;\widehat\mu\;\bigr= \mu, \, which means that the maximum likelihood estimator \widehat\mu is unbiased. Similarly we differentiate the log-likelihood with respect to and equate to zero: : \begin 0 & = \frac \log\Bigl( \mathcal (\mu,\sigma^2)\Bigr) = -\frac + \frac \sum_^ (\,x_i-\mu\,)^2. \end which is solved by :\widehat\sigma^2 = \frac \sum_^n(x_i-\mu)^2. Inserting the estimate \mu = \widehat\mu we obtain :\widehat\sigma^2 = \frac \sum_^n (x_i - \bar)^2 = \frac\sum_^n x_i^2 -\frac\sum_^n\sum_^n x_i x_j. To calculate its expected value, it is convenient to rewrite the expression in terms of zero-mean random variables ( statistical error) \delta_i \equiv \mu - x_i. Expressing the estimate in these variables yields : \widehat\sigma^2 = \frac \sum_^n (\mu - \delta_i)^2 -\frac\sum_^n\sum_^n (\mu - \delta_i)(\mu - \delta_j). Simplifying the expression above, utilizing the facts that \operatorname\bigl ;\delta_i\;\bigr= 0 and \operatorname\bigl ;\delta_i^2\;\bigr= \sigma^2 , allows us to obtain :\operatorname\bigl ;\widehat\sigma^2\;\bigr \frac\sigma^2. This means that the estimator \widehat\sigma^2 is biased for \sigma^2. It can also be shown that \widehat\sigma is biased for \sigma, but that both \widehat\sigma^2 and \widehat\sigma are consistent. Formally we say that the ''maximum likelihood estimator'' for \theta=(\mu,\sigma^2) is :\widehat = \left(\widehat,\widehat^2\right). In this case the MLEs could be obtained individually. In general this may not be the case, and the MLEs would have to be obtained simultaneously. The normal log-likelihood at its maximum takes a particularly simple form: : \log\Bigl( \mathcal(\widehat\mu,\widehat\sigma)\Bigr) = \frac \bigl(\,\log(2\pi\widehat\sigma^2) +1\,\bigr) This maximum log-likelihood can be shown to be the same for more general
least squares The method of least squares is a standard approach in regression analysis to approximate the solution of overdetermined systems (sets of equations in which there are more equations than unknowns) by minimizing the sum of the squares of the r ...
, even for non-linear least squares. This is often used in determining likelihood-based approximate
confidence interval In frequentist statistics, a confidence interval (CI) is a range of estimates for an unknown parameter. A confidence interval is computed at a designated ''confidence level''; the 95% confidence level is most common, but other levels, such as ...
s and
confidence region In statistics, a confidence region is a multi-dimensional generalization of a confidence interval. It is a set of points in an ''n''-dimensional space, often represented as an ellipsoid around a point which is an estimated solution to a problem, al ...
s, which are generally more accurate than those using the asymptotic normality discussed above.


Non-independent variables

It may be the case that variables are correlated, that is, not independent. Two random variables y_1 and y_2 are independent only if their joint probability density function is the product of the individual probability density functions, i.e. :f(y_1,y_2)=f(y_1)f(y_2)\, Suppose one constructs an order-''n'' Gaussian vector out of random variables (y_1,\ldots,y_n), where each variable has means given by (\mu_1, \ldots, \mu_n). Furthermore, let the covariance matrix be denoted by \mathit\Sigma. The joint probability density function of these ''n'' random variables then follows a
multivariate normal distribution In probability theory and statistics, the multivariate normal distribution, multivariate Gaussian distribution, or joint normal distribution is a generalization of the one-dimensional ( univariate) normal distribution to higher dimensions. One ...
given by: :f(y_1,\ldots,y_n)=\frac \exp\left( -\frac \left _1-\mu_1,\ldots,y_n-\mu_n\rightmathit\Sigma^ \left _1-\mu_1,\ldots,y_n-\mu_n\right\mathrm \right) In the bivariate case, the joint probability density function is given by: : f(y_1,y_2) = \frac \exp\left -\frac \left(\frac - \frac + \frac\right) \right In this and other cases where a joint density function exists, the likelihood function is defined as above, in the section " principles," using this density.


Example

X_1,\ X_2,\ldots,\ X_m are counts in cells / boxes 1 up to m; each box has a different probability (think of the boxes being bigger or smaller) and we fix the number of balls that fall to be n:x_1+x_2+\cdots+x_m=n. The probability of each box is p_i, with a constraint: p_1+p_2+\cdots+p_m=1. This is a case in which the X_i ''s'' are not independent, the joint probability of a vector x_1,\ x_2,\ldots,x_m is called the multinomial and has the form: : f(x_1,x_2,\ldots,x_m\mid p_1,p_2,\ldots,p_m)=\frac\prod p_i^= \binom p_1^ p_2^ \cdots p_m^ Each box taken separately against all the other boxes is a binomial and this is an extension thereof. The log-likelihood of this is: : \ell(p_1,p_2,\ldots,p_m)=\log n!-\sum_^m \log x_i!+\sum_^m x_i\log p_i The constraint has to be taken into account and use the Lagrange multipliers: : L(p_1,p_2,\ldots,p_m,\lambda)=\ell(p_1,p_2,\ldots,p_m)+\lambda\left(1-\sum_^m p_i\right) By posing all the derivatives to be 0, the most natural estimate is derived : \hat_i=\frac Maximizing log likelihood, with and without constraints, can be an unsolvable problem in closed form, then we have to use iterative procedures.


Iterative procedures

Except for special cases, the likelihood equations :\frac = 0 cannot be solved explicitly for an estimator \widehat = \widehat(\mathbf). Instead, they need to be solved iteratively: starting from an initial guess of \theta (say \widehat_), one seeks to obtain a convergent sequence \left\. Many methods for this kind of
optimization problem In mathematics, computer science and economics, an optimization problem is the problem of finding the ''best'' solution from all feasible solutions. Optimization problems can be divided into two categories, depending on whether the variables ...
are available, but the most commonly used ones are algorithms based on an updating formula of the form :\widehat_ = \widehat_ + \eta_ \mathbf_r\left(\widehat\right) where the vector \mathbf_\left(\widehat\right) indicates the descent direction of the rth "step," and the scalar \eta_ captures the "step length," also known as the
learning rate In machine learning and statistics, the learning rate is a Hyperparameter (machine learning), tuning parameter in an Mathematical optimization, optimization algorithm that determines the step size at each iteration while moving toward a minimum of ...
.


Gradient descent method

(Note: here it is a maximization problem, so the sign before gradient is flipped) :\eta_r\in \R^+ that is small enough for convergence and \mathbf_r\left(\widehat\right) = \nabla\ell\left(\widehat_r;\mathbf\right) Gradient descent method requires to calculate the gradient at the rth iteration, but no need to calculate the inverse of second-order derivative, i.e., the Hessian matrix. Therefore, it is computationally faster than Newton-Raphson method.


Newton–Raphson method

:\eta_r = 1 and \mathbf_r\left(\widehat\right) = -\mathbf^_r\left(\widehat\right) \mathbf_r\left(\widehat\right) where \mathbf_(\widehat) is the
score Score or scorer may refer to: *Test score, the result of an exam or test Business * Score Digital, now part of Bauer Radio * Score Entertainment, a former American trading card design and manufacturing company * Score Media, a former Canadian m ...
and \mathbf^_r \left(\widehat\right) is the
inverse Inverse or invert may refer to: Science and mathematics * Inverse (logic), a type of conditional sentence which is an immediate inference made from another conditional sentence * Additive inverse (negation), the inverse of a number that, when ad ...
of the Hessian matrix of the log-likelihood function, both evaluated the rth iteration. But because the calculation of the Hessian matrix is computationally costly, numerous alternatives have been proposed. The popular Berndt–Hall–Hall–Hausman algorithm approximates the Hessian with the
outer product In linear algebra, the outer product of two coordinate vectors is a matrix. If the two vectors have dimensions ''n'' and ''m'', then their outer product is an ''n'' × ''m'' matrix. More generally, given two tensors (multidimensional arrays of n ...
of the expected gradient, such that :\mathbf_r\left(\widehat\right) = - \left \frac \sum_^n \frac \left( \frac \right)^ \right \mathbf_r \left(\widehat\right)


Quasi-Newton methods

Other quasi-Newton methods use more elaborate secant updates to give approximation of Hessian matrix.


Davidon–Fletcher–Powell formula

DFP formula finds a solution that is symmetric, positive-definite and closest to the current approximate value of second-order derivative: : \mathbf_ = \left(I - \gamma_k y_k s_k^\mathsf\right) \mathbf_k \left(I - \gamma_k s_k y_k^\mathsf\right) + \gamma_k y_k y_k^\mathsf, where : y_k = \nabla\ell(x_k + s_k) - \nabla\ell(x_k), : \gamma_k = \frac, : s_k = x_ - x_k.


Broyden–Fletcher–Goldfarb–Shanno algorithm In numerical optimization, the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm is an iterative method for solving unconstrained nonlinear optimization problems. Like the related Davidon–Fletcher–Powell method, BFGS determines th ...

BFGS also gives a solution that is symmetric and positive-definite: : B_ = B_k + \frac - \frac\ , where : y_k = \nabla\ell(x_k + s_k) - \nabla\ell(x_k), : s_k = x_ - x_k. BFGS method is not guaranteed to converge unless the function has a quadratic
Taylor expansion In mathematics, the Taylor series or Taylor expansion of a function is an infinite sum of terms that are expressed in terms of the function's derivatives at a single point. For most common functions, the function and the sum of its Taylor se ...
near an optimum. However, BFGS can have acceptable performance even for non-smooth optimization instances


Fisher's scoring

Another popular method is to replace the Hessian with the Fisher information matrix, \mathcal(\theta) = \operatorname\left mathbf_r \left(\widehat\right)\right/math>, giving us the Fisher scoring algorithm. This procedure is standard in the estimation of many methods, such as generalized linear models. Although popular, quasi-Newton methods may converge to a
stationary point In mathematics, particularly in calculus, a stationary point of a differentiable function of one variable is a point on the graph of the function where the function's derivative is zero. Informally, it is a point where the function "stops" i ...
that is not necessarily a local or global maximum, but rather a local minimum or a saddle point. Therefore, it is important to assess the validity of the obtained solution to the likelihood equations, by verifying that the Hessian, evaluated at the solution, is both
negative definite In mathematics, negative definiteness is a property of any object to which a bilinear form may be naturally associated, which is negative-definite. See, in particular: * Negative-definite bilinear form * Negative-definite quadratic form * Negativ ...
and well-conditioned.


History

Early users of maximum likelihood were
Carl Friedrich Gauss Johann Carl Friedrich Gauss (; german: Gauß ; la, Carolus Fridericus Gauss; 30 April 177723 February 1855) was a German mathematician and physicist who made significant contributions to many fields in mathematics and science. Sometimes refe ...
,
Pierre-Simon Laplace Pierre-Simon, marquis de Laplace (; ; 23 March 1749 – 5 March 1827) was a French scholar and polymath whose work was important to the development of engineering, mathematics, statistics, physics, astronomy, and philosophy. He summarized ...
, Thorvald N. Thiele, and Francis Ysidro Edgeworth. However, its widespread use rose between 1912 and 1922 when
Ronald Fisher Sir Ronald Aylmer Fisher (17 February 1890 – 29 July 1962) was a British polymath who was active as a mathematician, statistician, biologist, geneticist, and academic. For his work in statistics, he has been described as "a genius who ...
recommended, widely popularized, and carefully analyzed maximum-likelihood estimation (with fruitless attempts at proofs). Maximum-likelihood estimation finally transcended heuristic justification in a proof published by Samuel S. Wilks in 1938, now called Wilks' theorem. The theorem shows that the error in the logarithm of likelihood values for estimates from multiple independent observations is asymptotically ''χ'' 2-distributed, which enables convenient determination of a
confidence region In statistics, a confidence region is a multi-dimensional generalization of a confidence interval. It is a set of points in an ''n''-dimensional space, often represented as an ellipsoid around a point which is an estimated solution to a problem, al ...
around any estimate of the parameters. The only difficult part of
Wilks Wilks is a surname which may refer to: People * Alan Wilks (born 1946), English footballer * Bobby Wilks (1931–2009), US Coast Guard aviator, first African-American to reach the rank of Coast Guard captain * Brian Wilks (born 1966), NHL hockey ...
’ proof depends on the expected value of the Fisher information matrix, which is provided by a theorem proven by Fisher. Wilks continued to improve on the generality of the theorem throughout his life, with his most general proof published in 1962. Reviews of the development of maximum likelihood estimation have been provided by a number of authors.


See also


Related concepts

*
Akaike information criterion The Akaike information criterion (AIC) is an estimator of prediction error and thereby relative quality of statistical models for a given set of data. Given a collection of models for the data, AIC estimates the quality of each model, relative to ...
: a criterion to compare statistical models, based on MLE * Extremum estimator: a more general class of estimators to which MLE belongs * Fisher information: information matrix, its relationship to covariance matrix of ML estimates *
Mean squared error In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference betwe ...
: a measure of how 'good' an estimator of a distributional parameter is (be it the maximum likelihood estimator or some other estimator) * RANSAC: a method to estimate parameters of a mathematical model given data that contains outliers * Rao–Blackwell theorem: yields a process for finding the best possible unbiased estimator (in the sense of having minimal
mean squared error In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference betwe ...
); the MLE is often a good starting place for the process * Wilks’ theorem: provides a means of estimating the size and shape of the region of roughly equally-probable estimates for the population's parameter values, using the information from a single sample, using a
chi-squared distribution In probability theory and statistics, the chi-squared distribution (also chi-square or \chi^2-distribution) with k degrees of freedom is the distribution of a sum of the squares of k independent standard normal random variables. The chi-squar ...


Other estimation methods

* Generalized method of moments: methods related to the likelihood equation in maximum likelihood estimation * M-estimator: an approach used in robust statistics * Maximum a posteriori (MAP) estimator: for a contrast in the way to calculate estimators when prior knowledge is postulated * Maximum spacing estimation: a related method that is more robust in many situations * Maximum entropy estimation * Method of moments (statistics): another popular method for finding parameters of distributions *
Method of support In statistics, the method of support is a technique that is used to make inferences from datasets. According to A. W. F. Edwards, the method of support aims to make inferences about unknown parameters in terms of the relative support, or log likel ...
, a variation of the maximum likelihood technique *
Minimum-distance estimation Minimum-distance estimation (MDE) is a conceptual method for fitting a statistical model to data, usually the empirical distribution. Often-used estimators such as ordinary least squares can be thought of as special cases of minimum-distance estim ...
* Partial likelihood methods for panel data * Quasi-maximum likelihood estimator: an MLE estimator that is misspecified, but still consistent * Restricted maximum likelihood: a variation using a likelihood function calculated from a transformed set of data


References


Further reading

* * * * * * * * *


External links

* * * * * {{DEFAULTSORT:Maximum likelihood M-estimators Probability distribution fitting