In
statistics
Statistics (from German: '' Statistik'', "description of a state, a country") is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of data. In applying statistics to a scientific, indust ...
, the Gauss–Markov theorem (or simply Gauss theorem for some authors) states that the
ordinary least squares
In statistics, ordinary least squares (OLS) is a type of linear least squares method for choosing the unknown parameters in a linear regression model (with fixed level-one effects of a linear function of a set of explanatory variables) by the ...
(OLS) estimator has the lowest
sampling variance within the
class
Class or The Class may refer to:
Common uses not otherwise categorized
* Class (biology), a taxonomic rank
* Class (knowledge representation), a collection of individuals or objects
* Class (philosophy), an analytical concept used differently ...
of
linear
Linearity is the property of a mathematical relationship ('' function'') that can be graphically represented as a straight line. Linearity is closely related to '' proportionality''. Examples in physics include rectilinear motion, the linear ...
unbiased estimator
In statistics, an estimator is a rule for calculating an estimate of a given quantity based on observed data: thus the rule (the estimator), the quantity of interest (the estimand) and its result (the estimate) are distinguished. For example, the ...
s, if the
errors
An error (from the Latin ''error'', meaning "wandering") is an action which is inaccurate or incorrect. In some usages, an error is synonymous with a mistake. The etymology derives from the Latin term 'errare', meaning 'to stray'.
In statistics ...
in the
linear regression model are
uncorrelated
In probability theory and statistics, two real-valued random variables, X, Y, are said to be uncorrelated if their covariance, \operatorname ,Y= \operatorname Y- \operatorname \operatorname /math>, is zero. If two variables are uncorrelated, ther ...
, have
equal variances and expectation value of zero. The errors do not need to be
normal, nor do they need to be
independent and identically distributed
In probability theory and statistics, a collection of random variables is independent and identically distributed if each random variable has the same probability distribution as the others and all are mutually independent. This property is usual ...
(only
uncorrelated
In probability theory and statistics, two real-valued random variables, X, Y, are said to be uncorrelated if their covariance, \operatorname ,Y= \operatorname Y- \operatorname \operatorname /math>, is zero. If two variables are uncorrelated, ther ...
with mean zero and
homoscedastic with finite variance). The requirement that the estimator be unbiased cannot be dropped, since biased estimators exist with lower variance. See, for example, the
James–Stein estimator
The James–Stein estimator is a biased estimator of the mean, \boldsymbol\theta, of (possibly) correlated Gaussian distributed random vectors Y = \ with unknown means \.
It arose sequentially in two main published papers, the earlier version ...
(which also drops linearity),
ridge regression
Ridge regression is a method of estimating the coefficients of multiple-regression models in scenarios where the independent variables are highly correlated. It has been used in many fields including econometrics, chemistry, and engineering. Also ...
, or simply any
degenerate estimator.
The theorem was named after
Carl Friedrich Gauss
Johann Carl Friedrich Gauss (; german: Gauß ; la, Carolus Fridericus Gauss; 30 April 177723 February 1855) was a German mathematician and physicist who made significant contributions to many fields in mathematics and science. Sometimes refer ...
and
Andrey Markov
Andrey Andreyevich Markov, first name also spelled "Andrei", in older works also spelled Markoff) (14 June 1856 – 20 July 1922) was a Russian mathematician best known for his work on stochastic processes. A primary subject of his research lat ...
, although Gauss' work significantly predates Markov's. But while Gauss derived the result under the assumption of independence and normality, Markov reduced the assumptions to the form stated above. A further generalization to
non-spherical errors was given by
Alexander Aitken
Alexander Craig "Alec" Aitken (1 April 1895 – 3 November 1967) was one of New Zealand's most eminent mathematicians. In a 1935 paper he introduced the concept of generalized least squares, along with now standard vector/matrix notation fo ...
.
Statement
Suppose we have in matrix notation,
:
expanding to,
:
where
are non-random but unobservable parameters,
are non-random and observable (called the "explanatory variables"),
are random, and so
are random. The random variables
are called the "disturbance", "noise" or simply "error" (will be contrasted with "residual" later in the article; see
errors and residuals in statistics
In statistics and optimization, errors and residuals are two closely related and easily confused measures of the deviation of an observed value of an element of a statistical sample from its " true value" (not necessarily observable). The er ...
). Note that to include a constant in the model above, one can choose to introduce the constant as a variable
with a newly introduced last column of X being unity i.e.,
for all
. Note that though
as sample responses, are observable, the following statements and arguments including assumptions, proofs and the others assume under the only condition of knowing
but not
The Gauss–Markov assumptions concern the set of error random variables,
:
*They have mean zero:
*They are
homoscedastic, that is all have the same finite variance:
for all
and
*Distinct error terms are uncorrelated:
A linear estimator of
is a linear combination
:
in which the coefficients
are not allowed to depend on the underlying coefficients
, since those are not observable, but are allowed to depend on the values
, since these data are observable. (The dependence of the coefficients on each
is typically nonlinear; the estimator is linear in each
and hence in each random
which is why this is
"linear" regression.) The estimator is said to be unbiased
if and only if
In logic and related fields such as mathematics and philosophy, "if and only if" (shortened as "iff") is a biconditional logical connective between statements, where either both statements are true or both are false.
The connective is bic ...
:
regardless of the values of
. Now, let
be some linear combination of the coefficients. Then the
mean squared error
In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference between ...
of the corresponding estimation is
:
in other words it is the expectation of the square of the weighted sum (across parameters) of the differences between the estimators and the corresponding parameters to be estimated. (Since we are considering the case in which all the parameter estimates are unbiased, this mean squared error is the same as the variance of the linear combination.) The best linear unbiased estimator (BLUE) of the vector
of parameters
is one with the smallest mean squared error for every vector
of linear combination parameters. This is equivalent to the condition that
:
is a positive semi-definite matrix for every other linear unbiased estimator
.
The ordinary least squares estimator (OLS) is the function
:
of
and
(where
denotes the
transpose
In linear algebra, the transpose of a matrix is an operator which flips a matrix over its diagonal;
that is, it switches the row and column indices of the matrix by producing another matrix, often denoted by (among other notations).
The tr ...
of
) that minimizes the sum of squares of
residuals (misprediction amounts):
:
The theorem now states that the OLS estimator is a BLUE. The main idea of the proof is that the least-squares estimator is uncorrelated with every linear unbiased estimator of zero, i.e., with every linear combination
whose coefficients do not depend upon the unobservable
but whose expected value is always zero.
Remark
Proof that the OLS indeed MINIMIZES the sum of squares of residuals may proceed as follows with a calculation of the
Hessian matrix
In mathematics, the Hessian matrix or Hessian is a square matrix of second-order partial derivatives of a scalar-valued function, or scalar field. It describes the local curvature of a function of many variables. The Hessian matrix was developed ...
and showing that it is positive definite.
The MSE function we want to minimize is
for a multiple regression model with ''p'' variables. The first derivative is
where ''X'' is the design matrix
The
Hessian matrix
In mathematics, the Hessian matrix or Hessian is a square matrix of second-order partial derivatives of a scalar-valued function, or scalar field. It describes the local curvature of a function of many variables. The Hessian matrix was developed ...
of second derivatives is
Assuming the columns of
are linearly independent so that
is invertible, let
, then
Now let
be an eigenvector of
.
In terms of vector multiplication, this means
where
is the eigenvalue corresponding to
. Moreover,
Finally, as eigenvector
was arbitrary, it means all eigenvalues of
are positive, therefore
is positive definite. Thus,
is indeed a global minimum.
Or, just see that for all vectors
. So the Hessian is positive definite if full rank.
Proof
Let
be another linear estimator of
with
where
is a
non-zero matrix. As we're restricting to ''unbiased'' estimators, minimum mean squared error implies minimum variance. The goal is therefore to show that such an estimator has a variance no smaller than that of
the OLS estimator. We calculate:
:
Therefore, since
is unobservable,
is unbiased if and only if
. Then:
:
Since ''DD is a positive semidefinite matrix,
exceeds
by a positive semidefinite matrix.
Remarks on the proof
As it has been stated before, the condition of
is a positive semidefinite matrix is equivalent to the property that the best linear unbiased estimator of
is
(best in the sense that it has minimum variance). To see this, let
another linear unbiased estimator of
.
:
Moreover, equality holds if and only if
. We calculate
:
This proves that the equality holds if and only if
which gives the uniqueness of the OLS estimator as a BLUE.
Generalized least squares estimator
The
generalized least squares
In statistics, generalized least squares (GLS) is a technique for estimating the unknown parameters in a linear regression model when there is a certain degree of correlation between the residuals in a regression model. In these cases, ordinar ...
(GLS), developed by
Aitken,
extends the Gauss–Markov theorem to the case where the error vector has a non-scalar covariance matrix.
The Aitken estimator is also a BLUE.
Gauss–Markov theorem as stated in econometrics
In most treatments of OLS, the regressors (parameters of interest) in the
design matrix are assumed to be fixed in repeated samples. This assumption is considered inappropriate for a predominantly nonexperimental science like
econometrics
Econometrics is the application of statistical methods to economic data in order to give empirical content to economic relationships. M. Hashem Pesaran (1987). "Econometrics," '' The New Palgrave: A Dictionary of Economics'', v. 2, p. 8 p. ...
. Instead, the assumptions of the Gauss–Markov theorem are stated conditional on
.
Linearity
The dependent variable is assumed to be a linear function of the variables specified in the model. The specification must be linear in its parameters. This does not mean that there must be a linear relationship between the independent and dependent variables. The independent variables can take non-linear forms as long as the parameters are linear. The equation
qualifies as linear while
can be transformed to be linear by replacing
by another parameter, say
. An equation with a parameter dependent on an independent variable does not qualify as linear, for example
, where
is a function of
.
Data transformations are often used to convert an equation into a linear form. For example, the
Cobb–Douglas function—often used in economics—is nonlinear:
:
But it can be expressed in linear form by taking the
natural logarithm
The natural logarithm of a number is its logarithm to the base of the mathematical constant , which is an irrational and transcendental number approximately equal to . The natural logarithm of is generally written as , , or sometimes, if ...
of both sides:
:
This assumption also covers specification issues: assuming that the proper functional form has been selected and there are no
omitted variables.
One should be aware, however, that the parameters that minimize the residuals of the transformed equation do not necessarily minimize the residuals of the original equation.
Strict exogeneity
For all
observations, the expectation—conditional on the regressors—of the error term is zero:
:
where
is the data vector of regressors for the ''i''th observation, and consequently
is the data matrix or design matrix.
Geometrically, this assumption implies that
and
are
orthogonal
In mathematics, orthogonality is the generalization of the geometric notion of '' perpendicularity''.
By extension, orthogonality is also used to refer to the separation of specific features of a system. The term also has specialized meanings in ...
to each other, so that their
inner product
In mathematics, an inner product space (or, rarely, a Hausdorff pre-Hilbert space) is a real vector space or a complex vector space with an operation called an inner product. The inner product of two vectors in the space is a scalar, often ...
(i.e., their cross moment) is zero.
:
This assumption is violated if the explanatory variables are
measured with error, or are
endogenous
Endogenous substances and processes are those that originate from within a living system such as an organism, tissue, or cell.
In contrast, exogenous substances and processes are those that originate from outside of an organism.
For example, ...
. Endogeneity can be the result of
simultaneity, where causality flows back and forth between both the dependent and independent variable.
Instrumental variable techniques are commonly used to address this problem.
Full rank
The sample data matrix
must have full column
rank
Rank is the relative position, value, worth, complexity, power, importance, authority, level, etc. of a person or object within a ranking, such as:
Level or position in a hierarchical organization
* Academic rank
* Diplomatic rank
* Hierarchy
* ...
.
:
Otherwise
is not invertible and the OLS estimator cannot be computed.
A violation of this assumption is
perfect multicollinearity, i.e. some explanatory variables are linearly dependent. One scenario in which this will occur is called "dummy variable trap," when a base dummy variable is not omitted resulting in perfect correlation between the dummy variables and the constant term.
Multicollinearity (as long as it is not "perfect") can be present resulting in a less efficient, but still unbiased estimate. The estimates will be less precise and highly sensitive to particular sets of data. Multicollinearity can be detected from
condition number
In numerical analysis, the condition number of a function measures how much the output value of the function can change for a small change in the input argument. This is used to measure how sensitive a function is to changes or errors in the inpu ...
or the
variance inflation factor
In statistics, the variance inflation factor (VIF) is the ratio (quotient) of the variance of estimating some parameter in a model that includes multiple other terms (parameters) by the variance of a model constructed using only one term. It quant ...
, among other tests.
Spherical errors
The
outer product
In linear algebra, the outer product of two coordinate vectors is a matrix. If the two vectors have dimensions ''n'' and ''m'', then their outer product is an ''n'' × ''m'' matrix. More generally, given two tensors (multidimensional arrays of nu ...
of the error vector must be spherical.
:
This implies the error term has uniform variance (
homoscedasticity
In statistics, a sequence (or a vector) of random variables is homoscedastic () if all its random variables have the same finite variance. This is also known as homogeneity of variance. The complementary notion is called heteroscedasticity. Th ...
) and no serial dependence. If this assumption is violated, OLS is still unbiased, but inefficient. The term "spherical errors" will describe the multivariate normal distribution: if
in the multivariate normal density, then the equation
is the formula for a
ball
A ball is a round object (usually spherical, but can sometimes be ovoid) with several uses. It is used in ball games, where the play of the game follows the state of the ball as it is hit, kicked or thrown by players. Balls can also be used f ...
centered at μ with radius σ in n-dimensional space.
Heteroskedasticity occurs when the amount of error is correlated with an independent variable. For example, in a regression on food expenditure and income, the error is correlated with income. Low income people generally spend a similar amount on food, while high income people may spend a very large amount or as little as low income people spend. Heteroskedastic can also be caused by changes in measurement practices. For example, as statistical offices improve their data, measurement error decreases, so the error term declines over time.
This assumption is violated when there is
autocorrelation
Autocorrelation, sometimes known as serial correlation in the discrete time case, is the correlation of a signal with a delayed copy of itself as a function of delay. Informally, it is the similarity between observations of a random variable ...
. Autocorrelation can be visualized on a data plot when a given observation is more likely to lie above a fitted line if adjacent observations also lie above the fitted regression line. Autocorrelation is common in time series data where a data series may experience "inertia." If a dependent variable takes a while to fully absorb a shock. Spatial autocorrelation can also occur geographic areas are likely to have similar errors. Autocorrelation may be the result of misspecification such as choosing the wrong functional form. In these cases, correcting the specification is one possible way to deal with autocorrelation.
In the presence of spherical errors, the generalized least squares estimator can be shown to be BLUE.
See also
*
Independent and identically distributed random variables
In probability theory and statistics, a collection of random variables is independent and identically distributed if each random variable has the same probability distribution as the others and all are mutually independent. This property is usu ...
*
Linear regression
In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables (also known as dependent and independent variables). The case of one explanatory variable is cal ...
*
Measurement uncertainty
In metrology, measurement uncertainty is the expression of the statistical dispersion of the values attributed to a measured quantity. All measurements are subject to uncertainty and a measurement result is complete only when it is accompanied by ...
Other unbiased statistics
*
Best linear unbiased prediction (BLUP)
*
Minimum-variance unbiased estimator (MVUE)
References
Further reading
*
*
*
External links
Earliest Known Uses of Some of the Words of Mathematics: G(brief history and explanation of the name)
(makes use of matrix algebra)
{{DEFAULTSORT:Gauss-Markov theorem
Theorems in statistics