In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of a probability distribution by maximizing a likelihood function, so that under the assumed statistical model the observed data is most probable. The point in the parameter space that maximizes the likelihood function is called the maximum likelihood estimate. The logic of maximum likelihood is both intuitive and flexible, and as such the method has become a dominant means of statistical inference.
If the likelihood function is differentiable, the derivative test for determining maxima can be applied. In some cases, the first-order conditions of the likelihood function can be solved explicitly; for instance, the ordinary least squares estimator maximizes the likelihood of the linear regression model. Under most circumstances, however, numerical methods will be necessary to find the maximum of the likelihood function.
From the vantage point of Bayesian inference, MLE is a special case of maximum a posteriori estimation (MAP) that assumes a uniform prior distribution of the parameters. In frequentist inference, MLE is a special case of an extremum estimator, with the objective function being the likelihood.

** Principles **

From a statistical standpoint, a given set of observations is a random sample from an unknown population. The goal of maximum likelihood estimation is to make inferences about the population that is most likely to have generated the sample, specifically the joint probability distribution of the random variables $\backslash left\backslash $, not necessarily independent and identically distributed. Associated with each probability distribution is a unique vector $\backslash theta\; =\; \backslash left\backslash theta\_,\backslash ,\; \backslash theta\_2,\backslash ,\; \backslash ldots,\backslash ,\; \backslash theta\_k\; \backslash right$ of parameters that index the probability distribution within a parametric family $\backslash $, where $\backslash Theta$ is called the parameter space, a finite-dimensional subset of Euclidean space. Evaluating the joint density at the observed data sample $\backslash mathbf\; =\; (y\_1,\; y\_2,\; \backslash ldots,\; y\_n)$ gives a real-valued function,
:$L\_(\backslash theta)\; =\; L\_(\backslash theta;\; \backslash mathbf)\; =\; f\_(\backslash mathbf;\; \backslash theta)$
which is called the likelihood function. For independent and identically distributed random variables, $f\_(\backslash mathbf;\; \backslash theta)$ will be the product of univariate density functions.
The goal of maximum likelihood estimation is to find the values of the model parameters that maximize the likelihood function over the parameter space, that is
:$\backslash hat\; =\; \backslash underset\backslash ,\backslash widehat\; L\_(\backslash theta\backslash ,;\backslash mathbf)$
Intuitively, this selects the parameter values that make the observed data most probable. The specific value $\backslash hat\; =\; \backslash hat\_(\backslash mathbf)\; \backslash in\; \backslash Theta$ that maximizes the likelihood function $L\_$ is called the maximum likelihood estimate. Further, if the function $\backslash hat\_\; :\; \backslash mathbb^\; \backslash to\; \backslash Theta$ so defined is measurable, then it is called the maximum likelihood estimator. It is generally a function defined over the sample space, i.e. taking a given sample as its argument. A sufficient but not necessary condition for its existence is for the likelihood function to be continuous over a parameter space $\backslash Theta$ that is compact. For an open $\backslash Theta$ the likelihood function may increase without ever reaching a supremum value.
In practice, it is often convenient to work with the natural logarithm of the likelihood function, called the log-likelihood:
: $\backslash ell(\backslash theta\backslash ,;\backslash mathbf)\; =\; \backslash ln\; L\_(\backslash theta\backslash ,;\backslash mathbf).$
Since the logarithm is a monotonic function, the maximum of $\backslash ell(\backslash theta\backslash ,;\backslash mathbf)$ occurs at the same value of $\backslash theta$ as does the maximum of $L\_$. If $\backslash ell(\backslash theta\backslash ,;\backslash mathbf)$ is differentiable in $\backslash theta$, the necessary conditions for the occurrence of a maximum (or a minimum) are
:$\backslash frac\; =\; 0,\; \backslash quad\; \backslash frac\; =\; 0,\; \backslash quad\; \backslash ldots,\; \backslash quad\; \backslash frac\; =\; 0,$
known as the likelihood equations. For some models, these equations can be explicitly solved for $\backslash widehat$, but in general no closed-form solution to the maximization problem is known or available, and an MLE can only be found via numerical optimization. Another problem is that in finite samples, there may exist multiple roots for the likelihood equations. Whether the identified root $\backslash widehat$ of the likelihood equations is indeed a (local) maximum depends on whether the matrix of second-order partial and cross-partial derivatives, the so-called Hessian matrix
:$\backslash mathbf\backslash left(\backslash widehat\backslash right)\; =\; \backslash begin\; \backslash left.\; \backslash frac\; \backslash \_\; \&\; \backslash left.\; \backslash frac\; \backslash \_\; \&\; \backslash dots\; \&\; \backslash left.\; \backslash frac\; \backslash \_\; \backslash \backslash \; \backslash left.\; \backslash frac\; \backslash \_\; \&\; \backslash left.\; \backslash frac\; \backslash \_\; \&\; \backslash dots\; \&\; \backslash left.\; \backslash frac\; \backslash \_\; \backslash \backslash \; \backslash vdots\; \&\; \backslash vdots\; \&\; \backslash ddots\; \&\; \backslash vdots\; \backslash \backslash \; \backslash left.\; \backslash frac\; \backslash \_\; \&\; \backslash left.\; \backslash frac\; \backslash \_\; \&\; \backslash dots\; \&\; \backslash left.\; \backslash frac\; \backslash \_\; \backslash end,$
is negative semi-definite at $\backslash widehat$, as this indicates local concavity. Conveniently, most common probability distributions—in particular the exponential family—are logarithmically concave.

** Restricted parameter space **

While the domain of the likelihood function—the parameter space—is generally a finite-dimensional subset of Euclidean space, additional restrictions sometimes need to be incorporated into the estimation process. The parameter space can be expressed as
:$\backslash Theta\; =\; \backslash left\backslash $,
where $h(\backslash theta)\; =\; \backslash lefth\_(\backslash theta),\; h\_(\backslash theta),\; \backslash ldots,\; h\_(\backslash theta)\; \backslash right/math>\; is\; avector-valued\; functionmapping$ \backslash mathbb^$into$ \backslash mathbb^$.\; Estimating\; the\; true\; parameter$ \backslash theta$belonging\; to$ \backslash Theta$then,\; as\; a\; practical\; matter,\; means\; to\; find\; the\; maximum\; of\; the\; likelihood\; function\; subject\; to\; theconstraint$ h(\backslash theta)\; =\; 0$.\; Theoretically,\; the\; most\; natural\; approach\; to\; thisconstrained\; optimizationproblem\; is\; the\; method\; of\; substitution,\; that\; is\; "filling\; out"\; the\; restrictions$ h\_,\; h\_,\; \backslash ldots,\; h\_$to\; a\; set$ h\_,\; h\_,\; \backslash ldots,\; h\_,\; h\_,\; \backslash ldots,\; h\_$in\; such\; a\; way\; that$ h^\; =\; \backslash lefth\_,\; h\_,\; \backslash ldots,\; h\_\; \backslash right/math>\; is\; aone-to-one\; functionfrom$ \backslash mathbb^$to\; itself,\; and\; reparameterize\; the\; likelihood\; function\; by\; setting$ \backslash phi\_\; =\; h\_(\backslash theta\_,\; \backslash theta\_,\; \backslash ldots,\; \backslash theta\_)$.Because\; of\; the\; invariance\; of\; the\; maximum\; likelihood\; estimator,\; the\; properties\; of\; the\; MLE\; apply\; to\; the\; restricted\; estimates\; also.\; For\; instance,\; in\; amultivariate\; normal\; distributionthecovariance\; matrix$ \backslash Sigma$must\; bepositive-definite;\; this\; restriction\; can\; be\; imposed\; by\; replacing$ \backslash Sigma\; =\; \backslash Gamma^\; \backslash Gamma$,\; where$ \backslash Gamma$is\; a\; realupper\; triangular\; matrixand$ \backslash Gamma^$is\; itstranspose.\; In\; practice,\; restrictions\; are\; usually\; imposed\; using\; the\; method\; of\; Lagrange\; which,\; given\; the\; constraints\; as\; defined\; above,\; leads\; to\; the\; restricted\; likelihood\; equations\; :$ \backslash frac\; -\; \backslash frac\; \backslash lambda\; =\; 0$and$ h(\backslash theta)\; =\; 0$,\; where$ \backslash lambda\; =\; \backslash left\backslash lambda\_,\; \backslash lambda\_,\; \backslash ldots,\; \backslash lambda\_\backslash right\backslash mathsf$is\; a\; column-vector\; ofLagrange\; multipliers\; and$ \backslash frac$is\; the$$`k × r` Jacobian matrix of partial derivatives. Naturally, if the constraints are nonbinding at the maximum, the Lagrange multipliers should be zero. This in turn allows for a statistical test of the "validity" of the constraint, known as the Lagrange multiplier test.

** Properties **

A maximum likelihood estimator is an extremum estimator obtained by maximizing, as a function of ''θ'', the objective function $\backslash widehat(\backslash theta\backslash ,;x)$. If the data are independent and identically distributed, then we have
: $\backslash widehat(\backslash theta\backslash ,;x)=\backslash frac1n\; \backslash sum\_^n\; \backslash ln\; f(x\_i\backslash mid\backslash theta),$
this being the sample analogue of the expected log-likelihood $\backslash ell(\backslash theta)\; =\; \backslash operatorname,\; \backslash ln\; f(x\_i\backslash mid\backslash theta)\; \backslash ,/math>,\; where\; this\; expectation\; is\; taken\; with\; respect\; to\; the\; true\; density.\; Maximum-likelihood\; estimators\; have\; no\; optimum\; properties\; for\; finite\; samples,\; in\; the\; sense\; that\; (when\; evaluated\; on\; finite\; samples)\; other\; estimators\; may\; have\; greater\; concentration\; around\; the\; true\; parameter-value.\; However,\; like\; other\; estimation\; methods,\; maximum\; likelihood\; estimation\; possesses\; a\; number\; of\; attractivelimiting\; properties:\; As\; the\; sample\; size\; increases\; to\; infinity,\; sequences\; of\; maximum\; likelihood\; estimators\; have\; these\; properties:\; *Consistency:\; the\; sequence\; of\; MLEs\; converges\; in\; probability\; to\; the\; value\; being\; estimated.\; *\; Functional\; Invariance:\; If$ \backslash hat$is\; the\; maximum\; likelihood\; estimator\; for$ \backslash theta$,\; and\; if$ g(\backslash theta)$is\; any\; transformation\; of$ \backslash theta$,\; then\; the\; maximum\; likelihood\; estimator\; for$ \backslash alpha\; =\; g(\backslash theta\; )$is$ \backslash hat\; =\; g(\backslash hat\; )$.\; *Efficiency,\; i.e.\; it\; achieves\; theCram\xe9r\u2013Rao\; lower\; boundwhen\; the\; sample\; size\; tends\; to\; infinity.\; This\; means\; that\; no\; consistent\; estimator\; has\; lower\; asymptoticmean\; squared\; errorthan\; the\; MLE\; (or\; other\; estimators\; attaining\; this\; bound),\; which\; also\; means\; that\; MLE\; hasasymptotic\; normality.\; *\; Second-order\; efficiency\; after\; correction\; for\; bias.$

** Consistency **

Under the conditions outlined below, the maximum likelihood estimator is consistent. The consistency means that if the data were generated by $f(\backslash cdot\backslash ,;\backslash theta\_0)$ and we have a sufficiently large number of observations ''n'', then it is possible to find the value of ''θ''_{0} with arbitrary precision. In mathematical terms this means that as ''n'' goes to infinity the estimator $\backslash widehat$ converges in probability to its true value:
: $\backslash widehat\_\backslash mathrm\backslash \; \backslash xrightarrow\backslash \; \backslash theta\_0.$
Under slightly stronger conditions, the estimator converges almost surely (or ''strongly''):
: $\backslash widehat\_\backslash mathrm\backslash \; \backslash xrightarrow\backslash \; \backslash theta\_0.$
In practical applications, data is never generated by $f(\backslash cdot\backslash ,;\backslash theta\_0)$. Rather, $f(\backslash cdot\backslash ,;\backslash theta\_0)$ is a model, often in idealized form, of the process that generated by the data. It is a common aphorism in statistics that ''all models are wrong''. Thus, true consistency does not occur in practical applications. Nevertheless, consistency is often considered to be a desirable property for an estimator to have.
To establish consistency, the following conditions are sufficient.
The dominance condition can be employed in the case of i.i.d. observations. In the non-i.i.d. case, the uniform convergence in probability can be checked by showing that the sequence $\backslash widehat(\backslash theta\backslash mid\; x)$ is stochastically equicontinuous.
If one wants to demonstrate that the ML estimator $\backslash widehat$ converges to ''θ''_{0} almost surely, then a stronger condition of uniform convergence almost surely has to be imposed:
: $\backslash sup\_\; \backslash left\backslash |\backslash ;\backslash widehat(\backslash theta\backslash mid\; x)\; -\; \backslash ell(\backslash theta)\backslash ;\backslash right\backslash |\; \backslash \; \backslash xrightarrow\backslash \; 0.$
Additionally, if (as assumed above) the data were generated by $f(\backslash cdot\backslash ,;\backslash theta\_0)$, then under certain conditions, it can also be shown that the maximum likelihood estimator converges in distribution to a normal distribution. Specifically,By Theorem 3.3 in
: $\backslash sqrt\backslash left(\backslash widehat\_\backslash mathrm\; -\; \backslash theta\_0\backslash right)\backslash \; \backslash xrightarrow\backslash \; \backslash mathcal\backslash left(0,\backslash ,\; I^\backslash right)$
where is the Fisher information matrix.

** Functional invariance **

The maximum likelihood estimator selects the parameter value which gives the observed data the largest possible probability (or probability density, in the continuous case). If the parameter consists of a number of components, then we define their separate maximum likelihood estimators, as the corresponding component of the MLE of the complete parameter. Consistent with this, if $\backslash widehat$ is the MLE for $\backslash theta$, and if $g(\backslash theta)$ is any transformation of $\backslash theta$, then the MLE for $\backslash alpha=g(\backslash theta)$ is by definition
:$\backslash widehat\; =\; g(\backslash ,\backslash widehat\backslash ,).\; \backslash ,$
It maximizes the so-called profile likelihood:
:$\backslash bar(\backslash alpha)\; =\; \backslash sup\_\; L(\backslash theta).\; \backslash ,$
The MLE is also invariant with respect to certain transformations of the data. If $y=g(x)$ where $g$ is one to one and does not depend on the parameters to be estimated, then the density functions satisfy
:$f\_Y(y)\; =\; \backslash frac$
and hence the likelihood functions for $X$ and $Y$ differ only by a factor that does not depend on the model parameters.
For example, the MLE parameters of the log-normal distribution are the same as those of the normal distribution fitted to the logarithm of the data.

** Efficiency **

As assumed above, the data were generated by $~f(\backslash cdot\backslash ,;\backslash theta\_0)~,$ then under certain conditions, it can also be shown that the maximum likelihood estimator converges in distribution to a normal distribution. It is -consistent and asymptotically efficient, meaning that it reaches the Cramér–Rao bound. Specifically,
: $\backslash sqrt\; \backslash ,\; \backslash left(\; \backslash widehat\_\backslash text\; -\; \backslash theta\_0\; \backslash right)\backslash \; \backslash \; \backslash xrightarrow\backslash \; \backslash \; \backslash mathcal\; \backslash left(\; 0,\backslash \; \backslash mathcal^\; \backslash right)\; ~,$
where $~\backslash mathcal~$ is the Fisher information matrix:
: $\backslash mathcal\_\; =\; \backslash operatorname\; \backslash ,\; \backslash biggl\backslash ;\; -\; \backslash ;\; \backslash biggr~.$
In particular, it means that the bias of the maximum likelihood estimator is equal to zero up to the order .

** Second-order efficiency after correction for bias **

However, when we consider the higher-order terms in the expansion of the distribution of this estimator, it turns out that has bias of order . This bias is equal to (componentwise)
:$b\_h\; \backslash ;\; \backslash equiv\; \backslash ;\; \backslash operatorname\; \backslash biggl\backslash ;\; \backslash left(\; \backslash widehat\backslash theta\_\backslash mathrm\; -\; \backslash theta\_0\; \backslash right)\_h\; \backslash ;\; \backslash biggr\backslash ;\; =\; \backslash ;\; \backslash frac\; \backslash ,\; \backslash sum\_^m\; \backslash ;\; \backslash mathcal^\; \backslash ;\; \backslash mathcal^\; \backslash left(\; \backslash frac\; \backslash ,\; K\_\; \backslash ;\; +\; \backslash ;\; J\_\; \backslash right)$
where $\backslash mathcal^$ (with superscripts) denotes the (''j,k'')-th component of the ''inverse'' Fisher information matrix $\backslash mathcal^$, and
:$\backslash frac\; \backslash ,\; K\_\; \backslash ;\; +\; \backslash ;\; J\_\; \backslash ;\; =\; \backslash ;\; \backslash operatorname\backslash ,\backslash biggl;\; \backslash frac12\; \backslash frac\; +\; \backslash frac\backslash ,\backslash frac\; \backslash ;\; \backslash biggr~\; .$
Using these formulae it is possible to estimate the second-order bias of the maximum likelihood estimator, and ''correct'' for that bias by subtracting it:
: $\backslash widehat^*\_\backslash text\; =\; \backslash widehat\_\backslash text\; -\; \backslash widehat\; ~\; .$
This estimator is unbiased up to the terms of order , and is called the bias-corrected maximum likelihood estimator.
This bias-corrected estimator is ''second-order efficient'' (at least within the curved exponential family), meaning that it has minimal mean squared error among all second-order bias-corrected estimators, up to the terms of the order . It is possible to continue this process, that is to derive the third-order bias-correction term, and so on. However the maximum likelihood estimator is ''not'' third-order efficient.

** Relation to Bayesian inference **

A maximum likelihood estimator coincides with the most probable Bayesian estimator given a uniform prior distribution on the parameters. Indeed, the maximum a posteriori estimate is the parameter that maximizes the probability of given the data, given by Bayes' theorem:
: $\backslash operatorname(\backslash theta\backslash mid\; x\_1,x\_2,\backslash ldots,x\_n)\; =\; \backslash frac$
where $\backslash operatorname(\backslash theta)$ is the prior distribution for the parameter and where $\backslash operatorname(x\_1,x\_2,\backslash ldots,x\_n)$ is the probability of the data averaged over all parameters. Since the denominator is independent of , the Bayesian estimator is obtained by maximizing $f(x\_1,x\_2,\backslash ldots,x\_n\backslash mid\backslash theta)\backslash operatorname(\backslash theta)$ with respect to . If we further assume that the prior $\backslash operatorname(\backslash theta)$ is a uniform distribution, the Bayesian estimator is obtained by maximizing the likelihood function $f(x\_1,x\_2,\backslash ldots,x\_n\backslash mid\backslash theta)$. Thus the Bayesian estimator coincides with the maximum likelihood estimator for a uniform prior distribution $\backslash operatorname(\backslash theta)$.

** Application of maximum-likelihood estimation in Bayes decision theory **

In many practical applications in machine learning, maximum-likelihood estimation is used as the model for parameter estimation.
The Bayesian Decision theory is about designing a classifier that minimizes total expected risk, especially, when the costs (the loss function) associated with different decisions are equal, the classifier is minimizing the error over the whole distribution.
Thus, the Bayes Decision Rule is stated as
:"decide $\backslash ;w\_1\backslash ;$ if $~\backslash operatorname(w\_1|x)\; \backslash ;\; >\; \backslash ;\; \backslash operatorname(w\_2|x)~;~$ otherwise decide $\backslash ;w\_2\backslash ;$"
where $\backslash ;w\_1\backslash ,,\; w\_2\backslash ;$ are predictions of different classes. From a perspective of minimizing error, it can also be stated as
:$w\; =\; \backslash underset\; \backslash ;\; \backslash int\_^\backslash infty\; \backslash operatorname(\backslash text\backslash mid\; x)\backslash operatorname(x)\backslash ,\backslash operatornamex~$
where
:$\backslash operatorname(\backslash text\backslash mid\; x)\; =\; \backslash operatorname(w\_1\backslash mid\; x)~$
if we decide $\backslash ;w\_2\backslash ;$ and $\backslash ;\backslash operatorname(\backslash text\backslash mid\; x)\; =\; \backslash operatorname(w\_2|x)\backslash ;$ if we decide $\backslash ;w\_1\backslash ;.$
By applying Bayes' theorem
:$\backslash operatorname(w\_i\; \backslash mid\; x)\; =\; \backslash frac$,
and if we further assume the zero-or-one loss function, which is a same loss for all errors, the Bayes Decision rule can be reformulated as:
:$h\_\backslash text\; =\; \backslash underset\; \backslash ,\; \backslash bigl,\; \backslash operatorname(x\backslash mid\; w)\backslash ,\backslash operatorname(w)\; \backslash ,\backslash bigr;,$
where $h\_\backslash text$ is the prediction and $\backslash ;\backslash operatorname(w)\backslash ;$ is the prior probability.

** Relation to minimizing Kullback–Leibler divergence and cross entropy **

Finding $\backslash hat\; \backslash theta$ that maximizes the likelihood is asymptotically equivalent to finding the $\backslash hat\; \backslash theta$ that defines a probability distribution ($Q\_$) that has a minimal distance, in terms of Kullback–Leibler divergence, to the real probability distribution from which our data was generated (i.e., generated by $P\_$). In an ideal world, P and Q are the same (and the only thing unknown is $\backslash theta$ that defines P), but even if they are not and the model we use is misspecified, still the MLE will give us the "closest" distribution (within the restriction of a model Q that depends on $\backslash hat\; \backslash theta$) to the real distribution $P\_$.
Since cross entropy is just Shannon's Entropy plus KL divergence, and since the Entropy of $P\_$ is constant, then the MLE is also asymptotically minimizing cross entropy.

** Examples **

** Discrete uniform distribution **

Consider a case where ''n'' tickets numbered from 1 to ''n'' are placed in a box and one is selected at random (''see uniform distribution''); thus, the sample size is 1. If ''n'' is unknown, then the maximum likelihood estimator $\backslash widehat$ of ''n'' is the number ''m'' on the drawn ticket. (The likelihood is 0 for ''n'' < ''m'', for ''n'' ≥ ''m'', and this is greatest when ''n'' = ''m''. Note that the maximum likelihood estimate of ''n'' occurs at the lower extreme of possible values , rather than somewhere in the "middle" of the range of possible values, which would result in less bias.) The expected value of the number ''m'' on the drawn ticket, and therefore the expected value of $\backslash widehat$, is (''n'' + 1)/2. As a result, with a sample size of 1, the maximum likelihood estimator for ''n'' will systematically underestimate ''n'' by (''n'' − 1)/2.

** Discrete distribution, finite parameter space **

Suppose one wishes to determine just how biased an unfair coin is. Call the probability of tossing a ‘head’ ''p''. The goal then becomes to determine ''p''.
Suppose the coin is tossed 80 times: i.e. the sample might be something like ''x''_{1} = H, ''x''_{2} = T, ..., ''x''_{80} = T, and the count of the number of heads "H" is observed.
The probability of tossing tails is 1 − ''p'' (so here ''p'' is ''θ'' above). Suppose the outcome is 49 heads and 31 tails, and suppose the coin was taken from a box containing three coins: one which gives heads with probability ''p'' = , one which gives heads with probability ''p'' = and another which gives heads with probability ''p'' = . The coins have lost their labels, so which one it was is unknown. Using maximum likelihood estimation, the coin that has the largest likelihood can be found, given the data that were observed. By using the probability mass function of the binomial distribution with sample size equal to 80, number successes equal to 49 but for different values of ''p'' (the "probability of success"), the likelihood function (defined below) takes one of three values:
:$\backslash begin\; \backslash operatorname\backslash bigl;\backslash mathrm\; =\; 49\; \backslash mid\; p=\backslash tfrac\backslash ;\backslash bigr\&\; =\; \backslash binom(\backslash tfrac)^(1-\backslash tfrac)^\; \backslash approx\; 0.000,\; \backslash \backslash pt\backslash operatorname\backslash bigl;\backslash mathrm\; =\; 49\; \backslash mid\; p=\backslash tfrac\backslash ;\backslash bigr\&\; =\; \backslash binom(\backslash tfrac)^(1-\backslash tfrac)^\; \backslash approx\; 0.012,\; \backslash \backslash pt\backslash operatorname\backslash bigl;\backslash mathrm\; =\; 49\; \backslash mid\; p=\backslash tfrac\backslash ;\backslash bigr\&\; =\; \backslash binom(\backslash tfrac)^(1-\backslash tfrac)^\; \backslash approx\; 0.054~.\; \backslash end$
The likelihood is maximized when = , and so this is the ''maximum likelihood estimate'' for .

** Discrete distribution, continuous parameter space **

Now suppose that there was only one coin but its could have been any value The likelihood function to be maximised is
:$L(p)\; =\; f\_D(\backslash mathrm\; =\; 49\; \backslash mid\; p)\; =\; \backslash binom\; p^(1\; -\; p)^~,$
and the maximisation is over all possible values
One way to maximize this function is by differentiating with respect to and setting to zero:
:$\backslash begin\; 0\; \&\; =\; \backslash frac\; \backslash left(\; \backslash binom\; p^(1-p)^\; \backslash right)~,\; \backslash \backslash pt0\; \&\; =\; 49\; p^(1-p)^\; -\; 31\; p^(1-p)^\; \backslash \backslash pt\&\; =\; p^(1-p)^\backslash left49\; (1-p)\; -\; 31\; p\; \backslash right\backslash \backslash pt\&\; =\; p^(1-p)^\backslash left49\; -\; 80\; p\; \backslash right.\; \backslash end$
This is a product of three terms. The first term is 0 when = 0. The second is 0 when = 1. The third is zero when = . The solution that maximizes the likelihood is clearly = (since = 0 and = 1 result in a likelihood of 0). Thus the ''maximum likelihood estimator'' for is .
This result is easily generalized by substituting a letter such as in the place of 49 to represent the observed number of 'successes' of our Bernoulli trials, and a letter such as in the place of 80 to represent the number of Bernoulli trials. Exactly the same calculation yields which is the maximum likelihood estimator for any sequence of Bernoulli trials resulting in 'successes'.

** Continuous distribution, continuous parameter space **

For the normal distribution $\backslash mathcal(\backslash mu,\; \backslash sigma^2)$ which has probability density function
:$f(x\backslash mid\; \backslash mu,\backslash sigma^2)\; =\; \backslash frac\; \backslash exp\backslash left(-\backslash frac\; \backslash right),$
the corresponding probability density function for a sample of independent identically distributed normal random variables (the likelihood) is
:$f(x\_1,\backslash ldots,x\_n\; \backslash mid\; \backslash mu,\backslash sigma^2)\; =\; \backslash prod\_^n\; f(\; x\_i\backslash mid\; \backslash mu,\; \backslash sigma^2)\; =\; \backslash left(\; \backslash frac\; \backslash right)^\; \backslash exp\backslash left(\; -\backslash frac\backslash right).$
This family of distributions has two parameters: ; so we maximize the likelihood, $\backslash mathcal\; (\backslash mu,\backslash sigma)\; =\; f(x\_1,\backslash ldots,x\_n\; \backslash mid\; \backslash mu,\; \backslash sigma)$, over both parameters simultaneously, or if possible, individually.
Since the logarithm function itself is a continuous strictly increasing function over the range of the likelihood, the values which maximize the likelihood will also maximize its logarithm (the log-likelihood itself is not necessarily strictly increasing). The log-likelihood can be written as follows:
:$\backslash log\backslash Bigl(\; \backslash mathcal\; (\backslash mu,\backslash sigma)\backslash Bigr)\; =\; -\backslash frac\; \backslash log(2\backslash pi\backslash sigma^2)\; -\; \backslash frac\; \backslash sum\_^n\; (\backslash ,x\_i-\backslash mu\backslash ,)^2$
(Note: the log-likelihood is closely related to information entropy and Fisher information.)
We now compute the derivatives of this log-likelihood as follows.
:$\backslash begin\; 0\; \&\; =\; \backslash frac\; \backslash log\backslash Bigl(\; \backslash mathcal\; (\backslash mu,\backslash sigma)\backslash Bigr)\; =\; 0\; -\; \backslash frac.\; \backslash end$
where $\backslash bar$ is the sample mean. This is solved by
:$\backslash widehat\backslash mu\; =\; \backslash bar\; =\; \backslash sum^n\_\; \backslash frac.$
This is indeed the maximum of the function, since it is the only turning point in and the second derivative is strictly less than zero. Its expected value is equal to the parameter of the given distribution,
:$\backslash operatorname\backslash bigl;\backslash widehat\backslash mu\backslash ;\backslash bigr=\; \backslash mu,\; \backslash ,$
which means that the maximum likelihood estimator $\backslash widehat\backslash mu$ is unbiased.
Similarly we differentiate the log-likelihood with respect to and equate to zero:
:$\backslash begin\; 0\; \&\; =\; \backslash frac\; \backslash log\backslash Bigl(\; \backslash mathcal\; (\backslash mu,\backslash sigma)\backslash Bigr)\; =\; -\backslash frac\; +\; \backslash frac\; \backslash sum\_^\; (\backslash ,x\_i-\backslash mu\backslash ,)^2.\; \backslash end$
which is solved by
:$\backslash widehat\backslash sigma^2\; =\; \backslash frac\; \backslash sum\_^n(x\_i-\backslash mu)^2.$
Inserting the estimate $\backslash mu\; =\; \backslash widehat\backslash mu$ we obtain
:$\backslash widehat\backslash sigma^2\; =\; \backslash frac\; \backslash sum\_^n\; (x\_i\; -\; \backslash bar)^2\; =\; \backslash frac\backslash sum\_^n\; x\_i^2\; -\backslash frac\backslash sum\_^n\backslash sum\_^n\; x\_i\; x\_j.$
To calculate its expected value, it is convenient to rewrite the expression in terms of zero-mean random variables (statistical error) $\backslash delta\_i\; \backslash equiv\; \backslash mu\; -\; x\_i$. Expressing the estimate in these variables yields
: $\backslash widehat\backslash sigma^2\; =\; \backslash frac\; \backslash sum\_^n\; (\backslash mu\; -\; \backslash delta\_i)^2\; -\backslash frac\backslash sum\_^n\backslash sum\_^n\; (\backslash mu\; -\; \backslash delta\_i)(\backslash mu\; -\; \backslash delta\_j).$
Simplifying the expression above, utilizing the facts that $\backslash operatorname\backslash bigl;\backslash delta\_i\backslash ;\backslash bigr=\; 0$ and $\backslash operatorname\backslash bigl;\backslash delta\_i^2\backslash ;\backslash bigr=\; \backslash sigma^2$, allows us to obtain
:$\backslash operatorname\backslash bigl;\backslash widehat\backslash sigma^2\backslash ;\backslash bigr\backslash frac\backslash sigma^2.$
This means that the estimator $\backslash widehat\backslash sigma$ is biased. However, $\backslash widehat\backslash sigma$ is consistent.
Formally we say that the ''maximum likelihood estimator'' for $\backslash theta=(\backslash mu,\backslash sigma^2)$ is
:$\backslash widehat\; =\; \backslash left(\backslash widehat,\backslash widehat^2\backslash right).$
In this case the MLEs could be obtained individually. In general this may not be the case, and the MLEs would have to be obtained simultaneously.
The normal log-likelihood at its maximum takes a particularly simple form:
:$\backslash log\backslash Bigl(\; \backslash mathcal(\backslash widehat\backslash mu,\backslash widehat\backslash sigma)\backslash Bigr)\; =\; \backslash frac\; \backslash bigl(\backslash ,\backslash log(2\backslash pi\backslash widehat\backslash sigma^2)\; +1\backslash ,\backslash bigr)$
This maximum log-likelihood can be shown to be the same for more general least squares, even for non-linear least squares. This is often used in determining likelihood-based approximate confidence intervals and confidence regions, which are generally more accurate than those using the asymptotic normality discussed above.

** Non-independent variables **

It may be the case that variables are correlated, that is, not independent. Two random variables $y\_1$ and $y\_2$ are independent only if their joint probability density function is the product of the individual probability density functions, i.e.
:$f(y\_1,y\_2)=f(y\_1)f(y\_2)\backslash ,$
Suppose one constructs an order-''n'' Gaussian vector out of random variables $(y\_1,\backslash ldots,y\_n)$, where each variable has means given by $(\backslash mu\_1,\; \backslash ldots,\; \backslash mu\_n)$. Furthermore, let the covariance matrix be denoted by $\backslash mathit\backslash Sigma$. The joint probability density function of these ''n'' random variables then follows a multivariate normal distribution given by:
:$f(y\_1,\backslash ldots,y\_n)=\backslash frac\; \backslash exp\backslash left(\; -\backslash frac\; \backslash left\_1-\backslash mu\_1,\backslash ldots,y\_n-\backslash mu\_n\backslash rightmathit\backslash Sigma^\; \backslash left\_1-\backslash mu\_1,\backslash ldots,y\_n-\backslash mu\_n\backslash right\backslash mathrm\; \backslash right)$
In the bivariate case, the joint probability density function is given by:
:$f(y\_1,y\_2)\; =\; \backslash frac\; \backslash exp\backslash left-\backslash frac\; \backslash left(\backslash frac\; -\; \backslash frac\; +\; \backslash frac\backslash right)\; \backslash right$
In this and other cases where a joint density function exists, the likelihood function is defined as above, in the section "principles," using this density.

** Example **

$X\_1,\backslash \; X\_2,\backslash ldots,\backslash \; X\_m$ are counts in cells / boxes 1 up to m; each box has a different probability (think of the boxes being bigger or smaller) and we fix the number of balls that fall to be $n$:$x\_1+x\_2+\backslash cdots+x\_m=n$. The probability of each box is $p\_i$, with a constraint: $p\_1+p\_2+\backslash cdots+p\_m=1$. This is a case in which the $X\_i$ ''s'' are not independent, the joint probability of a vector $x\_1,\backslash \; x\_2,\backslash ldots,x\_m$ is called the multinomial and has the form:
: $f(x\_1,x\_2,\backslash ldots,x\_m\backslash mid\; p\_1,p\_2,\backslash ldots,p\_m)=\backslash frac\backslash Pi\; p\_i^=\; \backslash binom\; p\_1^\; p\_2^\; \backslash cdots\; p\_m^$
Each box taken separately against all the other boxes is a binomial and this is an extension thereof.
The log-likelihood of this is:
: $\backslash ell(p\_1,p\_2,\backslash ldots,p\_m)=\backslash log\; n!-\backslash sum\_^m\; \backslash log\; x\_i!+\backslash sum\_^m\; x\_i\backslash log\; p\_i$
The constraint has to be taken into account and use the Lagrange multipliers:
: $L(p\_1,p\_2,\backslash ldots,p\_m,\backslash lambda)=\backslash ell(p\_1,p\_2,\backslash ldots,p\_m)+\backslash lambda\backslash left(1-\backslash sum\_^m\; p\_i\backslash right)$
By posing all the derivatives to be 0, the most natural estimate is derived
: $\backslash hat\_i=\backslash frac$
Maximizing log likelihood, with and without constraints, can be an unsolvable problem in closed form, then we have to use iterative procedures.

** Iterative procedures **

Except for special cases, the likelihood equations
:$\backslash frac\; =\; 0$
cannot be solved explicitly for an estimator $\backslash widehat\; =\; \backslash widehat(\backslash mathbf)$. Instead, they need to be solved iteratively: starting from an initial guess of $\backslash theta$ (say $\backslash widehat\_$), one seeks to obtain a convergent sequence $\backslash left\backslash $. Many methods for this kind of optimization problem are available, but the most commonly used ones are algorithms based on an updating formula of the form
:$\backslash widehat\_\; =\; \backslash widehat\_\; +\; \backslash eta\_\; \backslash mathbf\_r\backslash left(\backslash widehat\backslash right)$
where the vector $\backslash mathbf\_\backslash left(\backslash widehat\backslash right)$ indicates the descent direction of the `r`th "step," and the scalar $\backslash eta\_$ captures the "step length," also known as the learning rate.

** Gradient descent method **

(Note: here it is a maximization problem, so the sign before gradient is flipped)
:$\backslash eta\_r\backslash in\; \backslash R^+$ that is small enough for convergence and $\backslash mathbf\_r\backslash left(\backslash widehat\backslash right)\; =\; \backslash nabla\backslash ell\backslash left(\backslash widehat\_r;\backslash mathbf\backslash right)$
Gradient descent method requires to calculate the gradient at the rth iteration, but no need to calculate the inverse of second-order derivative, i.e., the Hessian matrix. Therefore, it is computationally faster than Newton-Raphson method.

** Newton–Raphson method **

:$\backslash eta\_r\; =\; 1$ and $\backslash mathbf\_r\backslash left(\backslash widehat\backslash right)\; =\; -\backslash mathbf^\_r\backslash left(\backslash widehat\backslash right)\; \backslash mathbf\_r\backslash left(\backslash widehat\backslash right)$
where $\backslash mathbf\_(\backslash widehat)$ is the score and $\backslash mathbf^\_r\; \backslash left(\backslash widehat\backslash right)$ is the inverse of the Hessian matrix of the log-likelihood function, both evaluated the `r`th iteration. But because the calculation of the Hessian matrix is computationally costly, numerous alternatives have been proposed. The popular Berndt–Hall–Hall–Hausman algorithm approximates the Hessian with the outer product of the expected gradient, such that
:$\backslash mathbf\_r\backslash left(\backslash widehat\backslash right)\; =\; -\; \backslash left\backslash frac\; \backslash sum\_^n\; \backslash frac\; \backslash left(\; \backslash frac\; \backslash right)^\; \backslash right\backslash mathbf\_r\; \backslash left(\backslash widehat\backslash right)$

** Quasi-Newton methods **

Other quasi-Newton methods use more elaborate secant updates to give approximation of Hessian matrix.

** Davidon–Fletcher–Powell formula **

DFP formula finds a solution that is symmetric, positive-definite and closest to the current approximate value of second-order derivative:
: $\backslash mathbf\_\; =\; \backslash left(I\; -\; \backslash gamma\_k\; y\_k\; s\_k^\backslash mathsf\backslash right)\; \backslash mathbf\_k\; \backslash left(I\; -\; \backslash gamma\_k\; s\_k\; y\_k^\backslash mathsf\backslash right)\; +\; \backslash gamma\_k\; y\_k\; y\_k^\backslash mathsf,$
where
: $y\_k\; =\; \backslash nabla\backslash ell(x\_k\; +\; s\_k)\; -\; \backslash nabla\backslash ell(x\_k),$
: $\backslash gamma\_k\; =\; \backslash frac,$
: $s\_k\; =\; x\_\; -\; x\_k.$

** Broyden–Fletcher–Goldfarb–Shanno algorithm **

BFGS also gives a solution that is symmetric and positive-definite:
: $B\_\; =\; B\_k\; +\; \backslash frac\; -\; \backslash frac\backslash \; ,$
where
: $y\_k\; =\; \backslash nabla\backslash ell(x\_k\; +\; s\_k)\; -\; \backslash nabla\backslash ell(x\_k),$
: $s\_k\; =\; x\_\; -\; x\_k.$
BFGS method is not guaranteed to converge unless the function has a quadratic Taylor expansion near an optimum. However, BFGS can have acceptable performance even for non-smooth optimization instances

** Fisher's scoring **

Another popular method is to replace the Hessian with the Fisher information matrix, $\backslash mathcal(\backslash theta)\; =\; \backslash operatorname\backslash leftmathbf\_r\; \backslash left(\backslash widehat\backslash right)\backslash right/math>,\; giving\; us\; the\; Fisher\; scoring\; algorithm.\; This\; procedure\; is\; standard\; in\; the\; estimation\; of\; many\; methods,\; such\; asgeneralized\; linear\; models.\; Although\; popular,\; quasi-Newton\; methods\; may\; converge\; to\; astationary\; pointthat\; is\; not\; necessarily\; a\; local\; or\; global\; maximum,\; but\; rather\; a\; local\; minimum\; or\; asaddle\; point.\; Therefore,\; it\; is\; important\; to\; assess\; the\; validity\; of\; the\; obtained\; solution\; to\; the\; likelihood\; equations,\; by\; verifying\; that\; the\; Hessian,\; evaluated\; at\; the\; solution,\; is\; bothnegative\; definiteandwell-conditioned.$

** History **

Early users of maximum likelihood were Carl Friedrich Gauss, Pierre-Simon Laplace, Thorvald N. Thiele, and Francis Ysidro Edgeworth. However, its widespread use rose between 1912 and 1922 when Ronald Fisher recommended, widely popularized, and carefully analyzed maximum-likelihood estimation (with fruitless attempts at proofs).
Maximum-likelihood estimation finally transcended heuristic justification in a proof published by Samuel S. Wilks in 1938, now called Wilks' theorem. The theorem shows that the error in the logarithm of likelihood values for estimates from multiple independent observations is asymptotically ''χ''^{ 2}-distributed, which enables convenient determination of a confidence region around any estimate of the parameters. The only difficult part of Wilks’ proof depends on the expected value of the Fisher information matrix, which is provided by a theorem proven by Fisher. Wilks continued to improve on the generality of the theorem throughout his life, with his most general proof published in 1962.
Reviews of the development of maximum likelihood estimation have been provided by a number of authors.

** See also **

** Related concepts**

* Akaike information criterion, a criterion to compare statistical models, based on MLE
* Extremum estimator, a more general class of estimators to which MLE belongs
* Fisher information, information matrix, its relationship to covariance matrix of ML estimates
* Mean squared error, a measure of how 'good' an estimator of a distributional parameter is (be it the maximum likelihood estimator or some other estimator)
* RANSAC, a method to estimate parameters of a mathematical model given data that contains outliers
* Rao–Blackwell theorem, which yields a process for finding the best possible unbiased estimator (in the sense of having minimal mean squared error); the MLE is often a good starting place for the process
* Wilks’ theorem provides a means of estimating the size and shape of the region of roughly equally-probable estimates for the population's parameter values, using the information from a single sample, using a chi-squared distribution

** Other estimation methods **

* Generalized method of moments are methods related to the likelihood equation in maximum likelihood estimation
* M-estimator, an approach used in robust statistics
* Maximum a posteriori (MAP) estimator, for a contrast in the way to calculate estimators when prior knowledge is postulated
* Maximum spacing estimation, a related method that is more robust in many situations
* Maximum entropy estimation
* Method of moments (statistics), another popular method for finding parameters of distributions
* Method of support, a variation of the maximum likelihood technique
* Minimum distance estimation
* Partial likelihood methods for panel data
* Quasi-maximum likelihood estimator, an MLE estimator that is misspecified, but still consistent
* Restricted maximum likelihood, a variation using a likelihood function calculated from a transformed set of data

** References **

** Further reading **

*
*
*
*
*
*
*
*
*

** External links **

*
*
*
*
*
{{Statistics
Category:M-estimators
Category:Probability distribution fitting