In
statistics, regression validation is the process of deciding whether the numerical results quantifying hypothesized relationships between variables, obtained from
regression analysis
In statistical modeling, regression analysis is a set of statistical processes for estimating the relationships between a dependent variable (often called the 'outcome' or 'response' variable, or a 'label' in machine learning parlance) and one ...
, are acceptable as descriptions of the data. The validation process can involve analyzing the
goodness of fit of the regression, analyzing whether the
regression residuals are random, and checking whether the model's predictive performance deteriorates substantially when applied to data that were not used in model estimation.
Goodness of fit
One measure of goodness of fit is the ''R''
2 (
coefficient of determination), which in ordinary least squares with an intercept ranges between 0 and 1. However, an ''R''
2 close to 1 does not guarantee that the model fits the data well: as
Anscombe's quartet shows, a high ''R''
2 can occur in the presence of misspecification of the functional form of a relationship or in the presence of outliers that distort the true relationship.
One problem with the ''R''
2 as a measure of model validity is that it can always be increased by adding more variables into the model, except in the unlikely event that the additional variables are exactly uncorrelated with the dependent variable in the data sample being used. This problem can be avoided by doing an
F-test
An ''F''-test is any statistical test in which the test statistic has an ''F''-distribution under the null hypothesis. It is most often used when comparing statistical models that have been fitted to a data set, in order to identify the model ...
of the statistical significance of the increase in the ''R''
2, or by instead using the
adjusted ''R''2.
Analysis of residuals
The
residuals from a fitted model are the differences between the responses observed at each combination of values of the
explanatory variables and the corresponding prediction of the response computed using the regression function. Mathematically, the definition of the residual for the ''i''
th observation in the
data set A data set (or dataset) is a collection of data. In the case of tabular data, a data set corresponds to one or more database tables, where every column of a table represents a particular variable, and each row corresponds to a given record of the ...
is written
:
with ''y
i'' denoting the ''i''
th response in the data set and ''x
i'' the vector of explanatory variables, each set at the corresponding values found in the ''i''
th observation in the data set.
If the model fit to the data were correct, the residuals would approximate the random errors that make the relationship between the explanatory variables and the response variable a statistical relationship. Therefore, if the residuals appear to behave randomly, it suggests that the model fits the data well. On the other hand, if non-random structure is evident in the residuals, it is a clear sign that the model fits the data poorly. The next section details the types of plots to use to test different aspects of a model and gives the correct interpretations of different results that could be observed for each type of plot.
Graphical analysis of residuals
A basic, though not quantitatively precise, way to check for problems that render a model inadequate is to conduct a visual examination of the residuals (the mispredictions of the data used in quantifying the model) to look for obvious deviations from randomness. If a visual examination suggests, for example, the possible presence of
heteroskedasticity
In statistics, a sequence (or a vector) of random variables is homoscedastic () if all its random variables have the same finite variance. This is also known as homogeneity of variance. The complementary notion is called heteroscedasticity. The s ...
(a relationship between the variance of the model errors and the size of an independent variable's observations), then statistical tests can be performed to confirm or reject this hunch; if it is confirmed, different modeling procedures are called for.
Different types of plots of the residuals from a fitted model provide information on the adequacy of different aspects of the model.
#sufficiency of the functional part of the model:
scatter plots of residuals versus predictors
#non-constant variation across the data:
scatter plots of residuals versus predictors; for data collected over time, also plots of residuals against time
#drift in the errors (data collected over time):
run chart
A run chart, also known as a run-sequence plot is a graph that displays observed data in a time sequence. Often, the data displayed represent some aspect of the output or performance of a manufacturing or other business process. It is therefore ...
s of the response and errors versus time
#independence of errors:
lag plot
#normality of errors:
histogram and
normal probability plot
The normal probability plot is a graphical technique to identify substantive departures from normality. This includes identifying outliers, skewness, kurtosis, a need for transformations, and mixtures. Normal probability plots are made of raw ...
Graphical methods have an advantage over numerical methods for model validation because they readily illustrate a broad range of complex aspects of the relationship between the model and the data.
Quantitative analysis of residuals
Numerical methods also play an important role in model validation. For example, the
lack-of-fit test for assessing the correctness of the functional part of the model can aid in interpreting a borderline residual plot. One common situation when numerical validation methods take precedence over graphical methods is when the number of
parameters
A parameter (), generally, is any characteristic that can help in defining or classifying a particular system (meaning an event, project, object, situation, etc.). That is, a parameter is an element of a system that is useful, or critical, when ...
being estimated is relatively close to the size of the data set. In this situation residual plots are often difficult to interpret due to constraints on the residuals imposed by the estimation of the unknown parameters. One area in which this typically happens is in optimization applications using
designed experiments.
Logistic regression
In statistics, the logistic model (or logit model) is a statistical model that models the probability of an event taking place by having the log-odds for the event be a linear combination of one or more independent variables. In regression a ...
with
binary data
Binary data is data whose unit can take on only two possible states. These are often labelled as 0 and 1 in accordance with the binary numeral system and Boolean algebra.
Binary data occurs in many different technical and scientific fields, wher ...
is another area in which graphical residual analysis can be difficult.
Serial correlation
Autocorrelation, sometimes known as serial correlation in the discrete time case, is the correlation of a signal with a delayed copy of itself as a function of delay. Informally, it is the similarity between observations of a random variable a ...
of the residuals can indicate model misspecification, and can be checked for with the
Durbin–Watson statistic. The problem of
heteroskedasticity
In statistics, a sequence (or a vector) of random variables is homoscedastic () if all its random variables have the same finite variance. This is also known as homogeneity of variance. The complementary notion is called heteroscedasticity. The s ...
can be checked for in any of
several ways.
Out-of-sample evaluation
Cross-validation is the process of assessing how the results of a statistical analysis will generalize to an independent data set. If the model has been estimated over some, but not all, of the available data, then the model using the estimated parameters can be used to predict the held-back data. If, for example, the out-of-sample
mean squared error
In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference between ...
, also known as the
mean squared prediction error
In statistics the mean squared prediction error or mean squared error of the predictions of a smoothing or curve fitting procedure is the expected value of the squared difference between the fitted values implied by the predictive function \wid ...
, is substantially higher than the in-sample mean square error, this is a sign of deficiency in the model.
A development in medical statistics is the use of out-of-sample cross validation techniques in meta-analysis. It forms the basis of the ''validation statistic, Vn'', which is used to test the statistical validity of meta-analysis summary estimates. Essentially it measures a type of normalized prediction error and its distribution is a linear combination of ''χ''
2 variables of degree 1.
See also
*
All models are wrong
*
Prediction interval
In statistical inference, specifically predictive inference, a prediction interval is an estimate of an interval in which a future observation will fall, with a certain probability, given what has already been observed. Prediction intervals are ...
*
Resampling (statistics)
In statistics, resampling is the creation of new samples based on one observed sample.
Resampling methods are:
# Permutation tests (also re-randomization tests)
# Bootstrapping
In general, bootstrapping usually refers to a self-starting proces ...
*
Statistical conclusion validity
Statistical conclusion validity is the degree to which conclusions about the relationship among variables based on the data are correct or "reasonable". This began as being solely about whether the statistical conclusion about the relationship of ...
*
Statistical model specification
*
Statistical model validation
*
Validity (statistics)
*
Coefficient of determination
*
Lack-of-fit sum of squares
*
Reduced chi-squared
In statistics, the reduced chi-square statistic is used extensively in goodness of fit testing. It is also known as mean squared weighted deviation (MSWD) in isotopic dating and variance of unit weight in the context of weighted least squares.
...
References
Further reading
*
* ; republished in 1997 by
University of Michigan Press
The University of Michigan Press is part of Michigan Publishing at the University of Michigan Library. It publishes 170 new titles each year in the humanities and social sciences. Titles from the press have earned numerous awards, including ...
External links
How can I tell if a model fits my data? (NIST)NIST/SEMATECH e-Handbook of Statistical MethodsModel Diagnostics(
Eberly College of Science)
{{NIST-PD
Validity (statistics)