In
statistics
Statistics (from German language, German: ''wikt:Statistik#German, Statistik'', "description of a State (polity), state, a country") is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of ...
, the explained sum of squares (ESS), alternatively known as the model sum of squares or sum of squares due to regression (SSR – not to be confused with the
residual sum of squares
In statistics, the residual sum of squares (RSS), also known as the sum of squared estimate of errors (SSE), is the sum of the squares of residuals (deviations predicted from actual empirical values of data). It is a measure of the discrepan ...
(RSS) or sum of squares of errors), is a quantity used in describing how well a model, often a
regression model
In statistical modeling, regression analysis is a set of statistical processes for estimating the relationships between a dependent variable (often called the 'outcome' or 'response' variable, or a 'label' in machine learning parlance) and one ...
, represents the data being modelled. In particular, the explained sum of squares measures how much variation there is in the modelled values and this is compared to the
total sum of squares
In statistical data analysis the total sum of squares (TSS or SST) is a quantity that appears as part of a standard way of presenting results of such analyses. For a set of observations, y_i, i\leq n, it is defined as the sum over all squared dif ...
(TSS), which measures how much variation there is in the observed data, and to the
residual sum of squares
In statistics, the residual sum of squares (RSS), also known as the sum of squared estimate of errors (SSE), is the sum of the squares of residuals (deviations predicted from actual empirical values of data). It is a measure of the discrepan ...
, which measures the variation in the error between the observed data and modelled values.
Definition
The explained sum of squares (ESS) is the sum of the squares of the deviations of the predicted values from the mean value of a response variable, in a standard
regression model
In statistical modeling, regression analysis is a set of statistical processes for estimating the relationships between a dependent variable (often called the 'outcome' or 'response' variable, or a 'label' in machine learning parlance) and one ...
— for example, , where ''y''
''i'' is the ''i''
th observation of the
response variable
Dependent and independent variables are variables in mathematical modeling, statistical modeling and experimental sciences. Dependent variables receive this name because, in an experiment, their values are studied under the supposition or deman ...
, ''x''
''ji'' is the ''i''
th observation of the ''j''
th explanatory variable
Dependent and independent variables are variables in mathematical modeling, statistical modeling and experimental sciences. Dependent variables receive this name because, in an experiment, their values are studied under the supposition or deman ...
, ''a'' and ''b''
''j'' are
coefficient
In mathematics, a coefficient is a multiplicative factor in some term of a polynomial, a series, or an expression; it is usually a number, but may be any expression (including variables such as , and ). When the coefficients are themselves var ...
s, ''i'' indexes the observations from 1 to ''n'', and ''ε''
''i'' is the ''i''
th value of the
error term In mathematics and statistics, an error term is an additive type of error. Common examples include:
* errors and residuals in statistics, e.g. in linear regression
* the error term in numerical integration
In analysis, numerical integration ...
. In general, the greater the ESS, the better the estimated model performs.
If
and
are the estimated
coefficient
In mathematics, a coefficient is a multiplicative factor in some term of a polynomial, a series, or an expression; it is usually a number, but may be any expression (including variables such as , and ). When the coefficients are themselves var ...
s, then
:
is the ''i''
th predicted value of the response variable. The ESS is then:
:
:where
the value estimated by the regression line .
In some cases (see below):
total sum of squares
In statistical data analysis the total sum of squares (TSS or SST) is a quantity that appears as part of a standard way of presenting results of such analyses. For a set of observations, y_i, i\leq n, it is defined as the sum over all squared dif ...
(TSS) = explained sum of squares (ESS) +
residual sum of squares
In statistics, the residual sum of squares (RSS), also known as the sum of squared estimate of errors (SSE), is the sum of the squares of residuals (deviations predicted from actual empirical values of data). It is a measure of the discrepan ...
(RSS).
Partitioning in simple linear regression
The following equality, stating that the total sum of squares (TSS) equals the residual sum of squares (=SSE : the sum of squared errors of prediction) plus the explained sum of squares (SSR :the sum of squares due to regression or explained sum of squares), is generally true in simple linear regression:
:
Simple derivation
:
Square both sides and sum over all ''i'':
:
Here is how the last term above is zero from
simple linear regression
In statistics, simple linear regression is a linear regression model with a single explanatory variable. That is, it concerns two-dimensional sample points with one independent variable and one dependent variable (conventionally, the ''x'' and ...
:
:
:
So,
:
:
Therefore,
:
Partitioning in the general ordinary least squares model
The general regression model with ''n'' observations and ''k'' explanators, the first of which is a constant unit vector whose coefficient is the regression intercept, is
:
where ''y'' is an ''n'' × 1 vector of dependent variable observations, each column of the ''n'' × ''k'' matrix ''X'' is a vector of observations on one of the ''k'' explanators,
is a ''k'' × 1 vector of true coefficients, and ''e'' is an ''n'' × 1 vector of the true underlying errors. The
ordinary least squares
In statistics, ordinary least squares (OLS) is a type of linear least squares method for choosing the unknown parameters in a linear regression model (with fixed level-one effects of a linear function of a set of explanatory variables) by the prin ...
estimator for
is
:
The residual vector
is
, so the residual sum of squares
is, after simplification,
:
Denote as
the constant vector all of whose elements are the sample mean
of the dependent variable values in the vector ''y''. Then the total sum of squares is
:
The explained sum of squares, defined as the sum of squared deviations of the predicted values from the observed mean of ''y'', is
:
Using
in this, and simplifying to obtain
, gives the result that ''TSS'' = ''ESS'' + ''RSS'' if and only if
. The left side of this is
times the sum of the elements of ''y'', and the right side is
times the sum of the elements of
, so the condition is that the sum of the elements of ''y'' equals the sum of the elements of
, or equivalently that the sum of the prediction errors (residuals)
is zero. This can be seen to be true by noting the well-known OLS property that the ''k'' × 1 vector
: since the first column of ''X'' is a vector of ones, the first element of this vector
is the sum of the residuals and is equal to zero. This proves that the condition holds for the result that ''TSS'' = ''ESS'' + ''RSS''.
In linear algebra terms, we have
,
,
.
The proof can be simplified by noting that
. The proof is as follows:
:
Thus,
:
which again gives the result that ''TSS'' = ''ESS'' + ''RSS'', since
.
See also
*
Sum of squares (statistics)
*
Lack-of-fit sum of squares In statistics, a sum of squares due to lack of fit, or more tersely a lack-of-fit sum of squares, is one of the components of a partition of the sum of squares of residuals in an analysis of variance, used in the numerator in an F-test of the nul ...
*
Fraction of variance unexplained
In statistics, the fraction of variance unexplained (FVU) in the context of a regression task is the fraction of variance of the regressand (dependent variable) ''Y'' which cannot be explained, i.e., which is not correctly predicted, by the e ...
Notes
References
* S. E. Maxwell and H. D. Delaney (1990), "Designing experiments and analyzing data: A model comparison perspective". Wadsworth. pp. 289–290.
* G. A. Milliken and D. E. Johnson (1984), "Analysis of messy data", Vol. I: Designed experiments. Van Nostrand Reinhold. pp. 146–151.
* B. G. Tabachnick and L. S. Fidell (2007), "Experimental design using ANOVA". Duxbury. p. 220.
* B. G. Tabachnick and L. S. Fidell (2007), "Using multivariate statistics", 5th ed. Pearson Education. pp. 217–218.
{{DEFAULTSORT:Explained Sum Of Squares
Least squares