Nonparametric regression is a category of
regression analysis in which the predictor does not take a predetermined form but is constructed according to information derived from the data. That is, no parametric form is assumed for the relationship between predictors and dependent variable. Nonparametric regression requires larger sample sizes than regression based on
parametric model
In statistics, a parametric model or parametric family or finite-dimensional model is a particular class of statistical models. Specifically, a parametric model is a family of probability distributions that has a finite number of parameters.
Def ...
s because the data must supply the model structure as well as the model estimates.
Definition
In nonparametric regression, we have random variables
and
and assume the following relationship:
:
where
is some deterministic function.
Linear regression
In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables (also known as dependent and independent variables). The case of one explanatory variable is call ...
is a restricted case of nonparametric regression where
is assumed to be affine.
Some authors use a slightly stronger assumption of additive noise:
:
where the random variable
is the `noise term', with mean 0.
Without the assumption that
belongs to a specific parametric family of functions it is impossible to get an unbiased estimate for
, however most estimators are
consistent under suitable conditions.
List of general-purpose nonparametric regression algorithms
This is a non-exhaustive list of non-parametric models for regression.
* nearest neighbors, see
nearest-neighbor interpolation and
k-nearest neighbors algorithm
*
regression trees
A decision tree is a decision support tool that uses a tree-like model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility. It is one way to display an algorithm that only contains condit ...
*
kernel regression
*
local regression
*
multivariate adaptive regression splines
*
smoothing splines
Smoothing splines are function estimates, \hat f(x), obtained from a set of noisy observations y_i of the target f(x_i), in order to balance a measure of goodness of fit of \hat f(x_i) to y_i with a derivative based measure of the smoothness of ...
Examples
Gaussian process regression or Kriging
In Gaussian process regression, also known as Kriging, a Gaussian prior is assumed for the regression curve. The errors are assumed to have a
multivariate normal distribution and the regression curve is estimated by its
posterior mode. The Gaussian prior may depend on unknown hyperparameters, which are usually estimated via
empirical Bayes.
The hyperparameters typically specify a prior covariance kernel. In case the kernel should also be inferred nonparametrically from the data, the
critical filter can be used.
Smoothing splines
Smoothing splines are function estimates, \hat f(x), obtained from a set of noisy observations y_i of the target f(x_i), in order to balance a measure of goodness of fit of \hat f(x_i) to y_i with a derivative based measure of the smoothness of ...
have an interpretation as the posterior mode of a Gaussian process regression.
Kernel regression
Kernel regression estimates the continuous dependent variable from a limited set of data points by
convolving the data points' locations with a
kernel function—approximately speaking, the kernel function specifies how to "blur" the influence of the data points so that their values can be used to predict the value for nearby locations.
Regression trees
Decision tree learning algorithms can be applied to learn to predict a dependent variable from data.
Although the original Classification And Regression Tree (CART) formulation applied only to predicting univariate data, the framework can be used to predict multivariate data, including time series.
See also
*
Lasso (statistics)
*
Local regression
*
Non-parametric statistics
*
Semiparametric regression
*
Isotonic regression
*
Multivariate adaptive regression splines
References
Further reading
*
*
*
*
*
External links
HyperNiche, software for nonparametric multiplicative regressionScale-adaptive nonparametric regression(with Matlab software).
{{statistics