Smoothing Spline

Smoothing splines are function estimates, $\hat f(x)$, obtained from a set of noisy observations $y_i$ of the target $f(x_i)$, in order to balance a measure of goodness of fit of $\hat f(x_i)$ to $y_i$ with a derivative based measure of the smoothness of $\hat f(x)$. They provide a means for smoothing noisy $x_i, y_i$ data. The most familiar example is the cubic smoothing spline, but there are many other possibilities, including for the case where $x$ is a vector quantity.

Cubic spline definition

Let $\$ be a set of observations, modeled by the relation $Y_i = f(x_i) + \epsilon_i$ where the $\epsilon_i$ are independent, zero mean random variables (usually assumed to have constant variance). The cubic smoothing spline estimate $\hat f$ of the function $f$ is defined to be the minimizer (over the class of twice differentiable functions) of
:$\sum_^n \^2 + \lambda \int \hat f''(x)^2 \,dx.$

Remarks:
* $\lambda \ge 0$ is a smoothing parameter, controlling the trade-off between fidelity to the data and roughness of the function estimate. This is often estimated by generalized cross-validation, or by restricted marginal likelihood (REML) which exploits the link between spline smoothing and Bayesian estimation (the smoothing penalty can be viewed as being induced by a prior on the $f$).
* The integral is often evaluated over the whole real line although it is also possible to restrict the range to that of $x_i$.
* As $\lambda\to 0$ (no smoothing), the smoothing spline converges to the interpolating spline.
* As $\lambda\to\infty$ (infinite smoothing), the roughness penalty becomes paramount and the estimate converges to a linear least squares estimate.
* The roughness penalty based on the second derivative is the most common in modern statistics literature, although the method can easily be adapted to penalties based on other derivatives.
* In early literature, with equally-spaced ordered