HOME

TheInfoList



OR:

In
statistics Statistics (from German language, German: ', "description of a State (polity), state, a country") is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of data. In applying statistics to a s ...
, polynomial regression is a form of regression analysis in which the relationship between the independent variable ''x'' and the
dependent variable A variable is considered dependent if it depends on (or is hypothesized to depend on) an independent variable. Dependent variables are studied under the supposition or demand that they depend, by some law or rule (e.g., by a mathematical functio ...
''y'' is modeled as a
polynomial In mathematics, a polynomial is a Expression (mathematics), mathematical expression consisting of indeterminate (variable), indeterminates (also called variable (mathematics), variables) and coefficients, that involves only the operations of addit ...
in ''x''. Polynomial regression fits a nonlinear relationship between the value of ''x'' and the corresponding conditional mean of ''y'', denoted E(''y'' , ''x''). Although polynomial regression fits a nonlinear model to the data, as a statistical estimation problem it is linear, in the sense that the regression function E(''y'' ,  ''x'') is linear in the unknown
parameter A parameter (), generally, is any characteristic that can help in defining or classifying a particular system (meaning an event, project, object, situation, etc.). That is, a parameter is an element of a system that is useful, or critical, when ...
s that are estimated from the
data Data ( , ) are a collection of discrete or continuous values that convey information, describing the quantity, quality, fact, statistics, other basic units of meaning, or simply sequences of symbols that may be further interpreted for ...
. Thus, polynomial regression is a special case of
linear regression In statistics, linear regression is a statistical model, model that estimates the relationship between a Scalar (mathematics), scalar response (dependent variable) and one or more explanatory variables (regressor or independent variable). A mode ...
. The explanatory (independent) variables resulting from the polynomial expansion of the "baseline" variables are known as higher-degree terms. Such variables are also used in
classification Classification is the activity of assigning objects to some pre-existing classes or categories. This is distinct from the task of establishing the classes themselves (for example through cluster analysis). Examples include diagnostic tests, identif ...
settings.


History

Polynomial regression models are usually fit using the method of least squares. The least-squares method minimizes the
variance In probability theory and statistics, variance is the expected value of the squared deviation from the mean of a random variable. The standard deviation (SD) is obtained as the square root of the variance. Variance is a measure of dispersion ...
of the
unbiased Bias is a disproportionate weight ''in favor of'' or ''against'' an idea or thing, usually in a way that is inaccurate, closed-minded, prejudicial, or unfair. Biases can be innate or learned. People may develop biases for or against an individ ...
estimators of the coefficients, under the conditions of the Gauss–Markov theorem. The least-squares method was published in 1805 by Legendre and in 1809 by
Gauss Johann Carl Friedrich Gauss (; ; ; 30 April 177723 February 1855) was a German mathematician, astronomer, Geodesy, geodesist, and physicist, who contributed to many fields in mathematics and science. He was director of the Göttingen Observat ...
. The first
design A design is the concept or proposal for an object, process, or system. The word ''design'' refers to something that is or has been intentionally created by a thinking agent, and is sometimes used to refer to the inherent nature of something ...
of an
experiment An experiment is a procedure carried out to support or refute a hypothesis, or determine the efficacy or likelihood of something previously untried. Experiments provide insight into cause-and-effect by demonstrating what outcome occurs whe ...
for polynomial regression appeared in an 1815 paper of Gergonne. In the twentieth century, polynomial regression played an important role in the development of regression analysis, with a greater emphasis on issues of
design A design is the concept or proposal for an object, process, or system. The word ''design'' refers to something that is or has been intentionally created by a thinking agent, and is sometimes used to refer to the inherent nature of something ...
and
inference Inferences are steps in logical reasoning, moving from premises to logical consequences; etymologically, the word '' infer'' means to "carry forward". Inference is theoretically traditionally divided into deduction and induction, a distinct ...
. More recently, the use of polynomial models has been complemented by other methods, with non-polynomial models having advantages for some classes of problems.


Definition and example

The goal of regression analysis is to model the expected value of a dependent variable ''y'' in terms of the value of an independent variable (or vector of independent variables) ''x''. In simple linear regression, the model : y = \beta_0 + \beta_1 x + \varepsilon, \, is used, where ε is an unobserved random error with mean zero conditioned on a scalar variable ''x''. In this model, for each unit increase in the value of ''x'', the conditional expectation of ''y'' increases by ''β''1 units. In many settings, such a linear relationship may not hold. For example, if we are modeling the yield of a chemical synthesis in terms of the temperature at which the synthesis takes place, we may find that the yield improves by increasing amounts for each unit increase in temperature. In this case, we might propose a quadratic model of the form : y = \beta_0 + \beta_1x + \beta_2 x^2 + \varepsilon. \, In this model, when the temperature is increased from ''x'' to ''x'' + 1 units, the expected yield changes by \beta_1+\beta_2(2x+ 1). (This can be seen by replacing ''x'' in this equation with ''x''+1 and subtracting the equation in ''x'' from the equation in ''x''+1.) For
infinitesimal In mathematics, an infinitesimal number is a non-zero quantity that is closer to 0 than any non-zero real number is. The word ''infinitesimal'' comes from a 17th-century Modern Latin coinage ''infinitesimus'', which originally referred to the " ...
changes in ''x'', the effect on ''y'' is given by the
total derivative In mathematics, the total derivative of a function at a point is the best linear approximation near this point of the function with respect to its arguments. Unlike partial derivatives, the total derivative approximates the function with res ...
with respect to ''x'': \beta_1+2\beta_2x. The fact that the change in yield depends on ''x'' is what makes the relationship between ''x'' and ''y'' nonlinear even though the model is linear in the parameters to be estimated. In general, we can model the expected value of ''y'' as an ''n''th degree polynomial, yielding the general polynomial regression model : y = \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x^3 + \cdots + \beta_n x^n + \varepsilon. \, Conveniently, these models are all linear from the point of view of
estimation Estimation (or estimating) is the process of finding an estimate or approximation, which is a value that is usable for some purpose even if input data may be incomplete, uncertain, or unstable. The value is nonetheless usable because it is d ...
, since the regression function is linear in terms of the unknown parameters ''β''0, ''β''1, .... Therefore, for least squares analysis, the computational and inferential problems of polynomial regression can be completely addressed using the techniques of multiple regression. This is done by treating ''x'', ''x''2, ... as being distinct independent variables in a multiple regression model.


Matrix form and calculation of estimates

The polynomial regression model :y_i \,=\, \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \cdots + \beta_m x_i^m + \varepsilon_i\ (i = 1, 2, \dots , n) can be expressed in matrix form in terms of a design matrix \mathbf, a response vector \vec y, a parameter vector \vec\beta, and a vector \vec\varepsilon of random errors. The ''i''-th row of \mathbf and \vec y will contain the ''x'' and ''y'' value for the ''i''-th data sample. Then the model can be written as a
system of linear equations In mathematics, a system of linear equations (or linear system) is a collection of two or more linear equations involving the same variable (math), variables. For example, : \begin 3x+2y-z=1\\ 2x-2y+4z=-2\\ -x+\fracy-z=0 \end is a system of th ...
: : \begin y_1\\ y_2\\ y_3 \\ \vdots \\ y_n \end= \begin 1 & x_1 & x_1^2 & \dots & x_1^m \\ 1 & x_2 & x_2^2 & \dots & x_2^m \\ 1 & x_3 & x_3^2 & \dots & x_3^m \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_n & x_n^2 & \dots & x_n^m \end \begin \beta_0\\ \beta_1\\ \beta_2\\ \vdots \\ \beta_m \end + \begin \varepsilon_1\\ \varepsilon_2\\ \varepsilon_3 \\ \vdots \\ \varepsilon_n \end, which when using pure matrix notation is written as : \vec y = \mathbf \vec \beta + \vec\varepsilon. \, The vector of estimated polynomial regression coefficients (using ordinary least squares
estimation Estimation (or estimating) is the process of finding an estimate or approximation, which is a value that is usable for some purpose even if input data may be incomplete, uncertain, or unstable. The value is nonetheless usable because it is d ...
) is : \widehat = (\mathbf^\mathsf \mathbf)^\; \mathbf^\mathsf \vec y, \, assuming ''m'' < ''n'' which is required for the matrix to be invertible; then since \mathbf is a Vandermonde matrix, the invertibility condition is guaranteed to hold if all the x_i values are distinct. This is the unique least-squares solution.


Expanded formulas

The above matrix equations explain the behavior of polynomial regression well. However, to physically implement polynomial regression for a set of xy point pairs, more detail is useful. The below matrix equations for polynomial coefficients are expanded from regression theory without derivation and easily implemented. \begin \sum_^nx_i^ & \sum_^nx_i^ & \sum_^nx_i^ & \cdots & \sum_^nx_i^ \\ \sum_^nx_i^ & \sum_^nx_i^ & \sum_^nx_i^ & \cdots & \sum_^nx_i^ \\ \sum_^nx_i^ & \sum_^nx_i^ & \sum_^nx_i^ & \cdots & \sum_^nx_i^ \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ \sum_^nx_i^ & \sum_^nx_i^ & \sum_^nx_i^ & \dots & \sum_^nx_i^ \\ \end \begin \beta_0 \\ \beta_1 \\ \beta_2 \\ \cdots \\ \beta_m \\ \end = \begin \sum_^ny_ix_i^0 \\ \sum_^ny_ix_i^1 \\ \sum_^ny_ix_i^2 \\ \cdots \\ \sum_^ny_ix_i^m \\ \end After solving the above
system of linear equations In mathematics, a system of linear equations (or linear system) is a collection of two or more linear equations involving the same variable (math), variables. For example, : \begin 3x+2y-z=1\\ 2x-2y+4z=-2\\ -x+\fracy-z=0 \end is a system of th ...
for \beta_0\text\beta_m, the regression polynomial may be constructed as follows: \begin &\qquad \widehat = \beta_0x^0 + \beta_1x^1 + \beta_2x^2 + \cdots + \beta_mx^m \\ &\qquad \\ &\qquad \text \\ &\qquad n = \text x_iy_i\text \\ &\qquad m = \text \\ &\qquad \beta_ = \textx^ \\ &\qquad \widehat =\text \end


Interpretation

Although polynomial regression is technically a special case of multiple linear regression, the interpretation of a fitted polynomial regression model requires a somewhat different perspective. It is often difficult to interpret the individual coefficients in a polynomial regression fit, since the underlying monomials can be highly correlated. For example, ''x'' and ''x''2 have correlation around 0.97 when x is uniformly distributed on the interval (0, 1). Although the correlation can be reduced by using
orthogonal polynomials In mathematics, an orthogonal polynomial sequence is a family of polynomials such that any two different polynomials in the sequence are orthogonal In mathematics, orthogonality (mathematics), orthogonality is the generalization of the geom ...
, it is generally more informative to consider the fitted regression function as a whole. Point-wise or simultaneous confidence bands can then be used to provide a sense of the uncertainty in the estimate of the regression function.


Alternative approaches

Polynomial regression is one example of regression analysis using basis functions to model a functional relationship between two quantities. More specifically, it replaces x \in \mathbb R^ in linear regression with polynomial basis \varphi (x) \in \mathbb R^, e.g. ,x\mathbin ,x,x^2,\ldots,x^d/math>. A drawback of polynomial bases is that the basis functions are "non-local", meaning that the fitted value of ''y'' at a given value ''x'' = ''x''0 depends strongly on data values with ''x'' far from ''x''0. In modern statistics, polynomial basis-functions are used along with new basis functions, such as splines, radial basis functions, and wavelets. These families of basis functions offer a more parsimonious fit for many types of data. The goal of polynomial regression is to model a non-linear relationship between the independent and dependent variables (technically, between the independent variable and the conditional mean of the dependent variable). This is similar to the goal of nonparametric regression, which aims to capture non-linear regression relationships. Therefore, non-parametric regression approaches such as smoothing can be useful alternatives to polynomial regression. Some of these methods make use of a localized form of classical polynomial regression. An advantage of traditional polynomial regression is that the inferential framework of multiple regression can be used (this also holds when using other families of basis functions such as splines). A final alternative is to use kernelized models such as support vector regression with a polynomial kernel. If residuals have unequal variance, a weighted least squares estimator may be used to account for that.


See also

*
Curve fitting Curve fitting is the process of constructing a curve, or mathematical function, that has the best fit to a series of data points, possibly subject to constraints. Curve fitting can involve either interpolation, where an exact fit to the data is ...
* Line regression * Local polynomial regression * Polynomial and rational function modeling * Polynomial interpolation * Response surface methodology * Smoothing spline


Notes

* Microsoft Excel makes use of polynomial regression when fitting a trendline to data points on an X Y scatter plot.


References

{{Least Squares and Regression Analysis


External links


Curve Fitting
PhET Phet may refer to: * Cyclone Phet, a cyclone formed in the Arabian Sea in 2010 * Amphetamine Amphetamine (contracted from Alpha and beta carbon, alpha-methylphenethylamine, methylphenethylamine) is a central nervous system (CNS) stimulant tha ...
Interactive simulations, University of Colorado at Boulder Regression analysis