View source for Nonparametric regression using kernel and spline methods


{| class="wikitable"
!Copyright notice <!-- don't remove! --> 
|-
| This article ''Nonparametric regression using kernel and spline methods'' was adapted from an original article by Jean D. Opsomer, which appeared in ''StatProb: The Encyclopedia Sponsored by Statistics and Probability Societies''. The original article ([<nowiki>http://statprob.com/encyclopedia/NonparametricRegressionUsingKernelAndSplineMethods3.html</nowiki> StatProb Source], Local Files: [[Media:nonparametric_regression_using_kernel_and_spline_methods.pdf|pdf]] | [[Media:nonparametric_regression_using_kernel_and_spline_methods.tex|tex]]) is copyrighted by the author(s), the article has been donated to ''Encyclopedia of Mathematics'', and its further issues are under ''Creative Commons Attribution Share-Alike License'. All pages from StatProb are contained in the [[:Category:Statprob|Category StatProb]].
|-
|}

{{MSC|62G08}}


<!-- \documentclass[10pt]{article} 
     \usepackage{amssymb} 
     \usepackage{amsmath} 
     \usepackage{amsfonts} 
     \usepackage{graphicx} 
     \usepackage{epsfig} 
     \usepackage{chicago} -->

$ \def\R{\mbox{\rlap{I}\hskip.03in R}} $
$ \def\Z{\mbox{\rlap{Z}\hskip.03in Z}} $
$ \newcommand{\law}{\stackrel{ {\cal L}}{\to}} $
$ \newcommand{\prob}{\stackrel{p}{\to}} $
$ \newcommand{\bm}[1]{ { {\mathbf #1}}} $
$ \def\bn#1{ '''#1'''} $
$ \def\qed{\par {\hfill \rule{3mm}{3mm}}} $
$ \newcommand{\E}{ {\rm E}} $
$ \newcommand{\Var}{ {\rm Var}} $
$ \newcommand{\eac}[1]{ ''et al.''~{{Cite|#1}}} $
<!-- \newtheorem{theorem}{Theorem} 
     \newtheorem{lemma}{Lemma} 
     \newtheorem{propos}{Proposition} 
     \newtheorem{assume}{\rlap{A}\hskip.06in } 
     \newtheorem{cor}{Corollary} 
     \begin{document} -->
<center>'''Nonparametric regression using kernel and spline methods'''</center>

<center> Jean D. Opsomer<ref> Department of Statistics, Colorado State University, Fort Collins, CO, USA. Email: jopsomer@stat.colostate.edu. </ref>

and

F. Jay Breidt <ref> Department of Statistics, Colorado State University, Fort Collins, CO, USA. Email: jbreidt@stat.colostate.edu </ref>

</center>

<!-- \maketitle -->

== The statistical model ==
<!-- \label{model} -->

When applying nonparametric regression methods, the researcher is interested in estimating the relationship
between one dependent variable, $Y$, and one or several covariates,
$X_1,\ldots,X_q$. We discuss here the situation with one covariate, $X$ (the case with multiple covariates is addressed in the references provided below). The relationship between $X$ and $Y$ can be expressed as the conditional expectation
$$ 
\E(Y|X=x) = f(x).
$$ Unlike in parametric regression, the shape of the function $f(\cdot)$ is
not restricted to belong to a specific parametric family such as polynomials.

This representation for the mean function is the key difference between parametric and
nonparametric regression, and the remaining aspects of the statistical model for
$(X,Y)$ are similar between both regression approaches. In particular, the random variable
$Y$ is often assumed to have a constant (conditional) variance,
$\Var(Y|X)=\sigma^2$, with $\sigma^2$ unknown. The constant variance and other common regression model assumptions, such as independence, can be relaxed
just as in parametric regression.

== Kernel methods ==
Suppose that we have a dataset available with observations $(x_1,y_1),\ldots,(x_n,y_n)$. A simple kernel-based estimator of $f(x)$ is the ''Nadaraya-Watson kernel regression'' estimator, defined as
$$
\hat{f}_h(x) = \frac{\sum_{i=1}^n K_h(x_i-x) y_i}{\sum_{i=1}^n K_h(x_i-x)},
\label{nadwat}
\tag{1}$$

with $K_h(\cdot)=K(\cdot/h)/h$ for some kernel function $K(\cdot)$ and bandwidth parameter $h>0$.
The function $K(\cdot)$ is usually a symmetric probability density and examples of commonly used kernel functions are the Gaussian kernel
$K(t)=(\sqrt{2\pi})^{-1} \exp(-t^2/2)$ and the ''Epanechnikov'' kernel $K(t) = \max\{\frac{3}{4} (1-t^2),0\}$.

Generally, the researcher is not interested in estimating the value of $f(\cdot)$ at a
single location $x$, but in estimating the curve over a range of values, say for all $x
\in [a_x,b_x]$. In principle, kernel regression requires computing (1) for any
value of interest. In practice, $\hat{f}_h(x)$ is calculated on a sufficiently fine grid of $x$-values and the curve is obtained by interpolation.

We used the subscript $h$ in $\hat{f}_h(x)$ in (1) to emphasize the fact that the bandwidth $h$ is the main determinant of the shape of the estimated regression, as demonstrated in Figure 1. When $h$ is small relative to the range of the data, the resulting fit can be highly variable and look "wiggly." When $h$ is chosen to be larger, this results in a less variable, more smooth fit, but it makes the estimator less responsive to local features in the data and introduces the possibility of bias in the estimator. Selecting a value for the bandwidth in such a way that it balances the variance with the potential bias is therefore a crucial decision for researchers who want to apply nonparametric regression on their data. Data-driven bandwidth selection methods are available in the literature, including in the references provided below.

<!--
 \begin{figure}
\includegraphics{bank_swallow.eps}
\caption{\small Dates (Julian days) of first sightings of bank swallows in Cayuga Lake
basin, with three kernel regressions using bandwidth values $h$ calculated as the range of
years multiplied by 0.05 ($--$), 0.2 (--) and 0.4 ($-\cdot$).}
\label{kernregex}
\end{figure}
 -->

<span id="Fig1">
[[File:bank_swallow.png|center|800px|'''Figure 1:'''  Dates (Julian days) of first sightings of bank swallows in Cayuga Lake basin, with three kernel regressions using bandwidth values $h$ calculated as the range of years multiplied by 0.05 ($--$), 0.2 (--) and 0.4 ($-\cdot$). ([[Media:bank_swallow.eps|eps]]) ]]
</span>


A class of kernel-based estimators that generalizes the Nadaraya-Watson estimator in (1) is referred to as ''local polynomial regression'' estimators. At each location $x$, the estimator $\hat{f}_h(x)$ is obtained as the estimated intercept, $\hat{\beta}_0$, in the weighted least squares fit of a polynomial of degree $p$,
$$ 
\min_{\bm{\beta}} \sum_{i=1}^n \left(y_i - \beta_0 + \beta_1 (x_i-x) + \cdots + \beta_p (x_i-x)^p\right) K_h(x_i-x).
$$
This estimator can be written explicitly in matrix notation as
$$
\hat{f}_h(x) = (1,0,\ldots,0) \left(\bm{X}_x^T \bm{W}_x \bm{X}_x
\right)^{-1} \bm{X}_x^T \bm{W}_x
\bm{Y},
\label{lprdef}
\tag{2}$$

where $\bm{Y}=(y_1,\ldots,y_n)^T$, $\bm{W}_x =
\mbox{diag}\{K_h(x_1-x),\ldots,K_h(x_n-x)\}$ and
$$ 
\bm{X}_x = \left[ \begin{array}{cccc}
1 & x_1-x & \cdots & (x_1-x)^p \\
\vdots & \vdots & & \vdots \\
1 & x_n-x & \cdots & (x_n-x)^p
\end{array} \right].
$$
It should be noted that the Nadaraya-Watson estimator (1) is a special case of the local polynomial regression estimator with $p=0$. In practice, the local linear ($p=1$) and local quadratic estimators ($p=2$) are frequently used.

An extensive literature on kernel regression and local polynomial regression exists, and their theoretical properties are well understood. Both kernel regression and local polynomial regression estimators are biased but consistent estimators of the unknown mean function, when that function is continuous and sufficiently smooth. For further information on these methods, we refer to reader to the monographs by {{cite|wan95}} and {{cite|fan96}}.

== Spline methods ==
<!-- \label{spline} -->

In the previous section, the unknown mean function was assumed to be ''locally'' well
approximated by a polynomial, which led to local polynomial regression. An alternative
approach is to represent the fit as a ''piecewise'' polynomial, with the pieces
connecting at points called ''knots''. Once the knots are selected,
such an estimator can be computed globally in a manner similar to that for a
parametrically specified mean function, as will be explained below. A fitted mean
function represented by a piecewise continuous curve only rarely provides a satisfactory
fit, however, so that usually the function and at least its first derivative are
constrained to be continuous everywhere, with only the second or higher derivatives allowed to be
discontinuous at the knots. For historical reasons, these constrained piecewise
polynomials are referred to as ''splines'', leading to the name ''spline regression''
or ''spline smoothing'' for this type of nonparametric regression.

Consider the following simple type of polynomial spline of degree $p$:
$$
\beta_0 + \beta_1 x + \cdots + \beta_p x^p + \sum_{k=1}^K \beta_{p+k} (x-\kappa_k)^p_+,
\label{regspline}
\tag{3}$$

where $p\geq 1$, $\kappa_1,\ldots,\kappa_K$ are the knots and
$(\cdot)^p_+ = \max\{(\cdot)^p,0\}$. Clearly, (3) has continuous
derivatives up to degree $(p-1)$, but the $p$th derivative can be discontinuous at the
knots. Model (3) is constructed as a linear combination of ''basis functions'' $1,
x, \ldots, x^p, (x-\kappa_1)^p_+,\ldots, (x-\kappa_K)^p_+$. This basis is
referred to as the ''truncated power basis''. A popular set of basis functions are the so-called
''B-splines''. Unlike the truncated power splines, the B-splines have compact support and are numerically more stable, but they span the same function space. In what follows, we will write $\psi_j(x), j=1,\ldots,J$ for a set of (generic) basis
functions used in fitting regression splines, and replace (3) by
$\beta_1 \psi_1(x) + \cdots +\beta_J \psi_J(x)$.

For fixed knots, a regression spline is linear in the unknown parameters $\bm{\beta} = (\beta_1,\ldots,\beta_{J})^T$ and can be fitted parametrically using least squares techniques. Under the homoskedastic model described in [[#The statistical model| Section 1]], the ''regression spline'' estimator for $f(x)$ is
obtained by solving
$$
\hat{\bm{\beta}} = \arg \min_{\bm{\beta}} \sum_{i=1}^n \left(y_i - \sum_{j=1}^J \beta_j \psi_j(x_i)\right)^2
\label{Bspline}
\tag{4}$$

and setting $\hat{f}(x) = \sum_{j=1}^J \hat{\beta}_j \psi_j(x)$. Since deviations from the parametric shape can only occur at the knots, the amount of
smoothing is determined by the degree of the basis and the location and number of knots. In practice, the degree is fixed (with $p=1, 2$ or 3 as common choices) and the knot locations are usually chosen to be equally-spaced over the range of the data or placed at
regularly spaced data quantiles. Hence, the number of knots $K$ is the only
remaining smoothing parameter for the spline regression estimator. As $K$ (and therefore $J$) is chosen to be larger, increasingly flexible estimators for $f(\cdot)$ are produced. This reduces the potential bias due to approximating the unknown mean function by a spline function, but increases the variability of the estimators.

The ''smoothing spline'' estimator is an important extension of the regression spline estimator. The smoothing spline estimator for $f(\cdot)$ for a set of data generated by the
statistical model described in [[#The statistical model| Section 1]] is defined as the minimizer of
$$
\sum_{i=1}^n (y_i - f(x_i))^2 + \lambda
\int_{a_x}^{b_x} (f^{(p)}(t))^2 dt,
\label{sspline}
\tag{5}$$

over the set of all functions $f(\cdot)$ with continuous
$(p-1)$th derivative and square integrable $p$th derivative, and
$\lambda >0$ is a constant determining the degree of smoothness of the estimator. Larger values of $\lambda$ correspond to smoother fits. The
choice $p=2$ leads to the popular ''cubic smoothing splines''. While not immediately obvious
from the definition, the function minimizing (5) is exactly equal to a special
type of regression spline with knots at each of the observation points
$x_1,\ldots,x_n$ (assuming each of the locations $x_i$ is unique).

Traditional regression spline fitting as in (4) is usually done using a
relatively small number of knots. By construction, smoothing splines use a large number of knots (typically, $n$ knots), but the smoothness of the function is controlled by a penalty term and the smoothing
parameter $\lambda$. The ''penalized spline'' estimator represents a compromise between these two approaches. It uses a moderate number of knots and puts a penalty on the coefficients of the basis functions. Specifically, a simple type of penalized spline estimator for $m(\cdot)$ is obtained by solving
$$
\hat{\bm{\beta}} = \arg \min_{\bm{\beta}} \sum_{i=1}^n \left(y_i - \sum_{j=1}^J \beta_j \psi_j(x_i)\right)^2 + \lambda
\sum_{j=1}^J
\beta_j^2
\label{Pspline}
\tag{6}$$

and setting $\hat{f}_{\lambda}(x) = \sum_{j=1}^J \hat{\beta}_j \psi_j(x)$ as for regression splines. Penalized splines combine the advantage of a parametric fitting method, as for regression
splines, with the flexible adjustment of the degree of smoothness as in smoothing splines. Both the basis function and the exact form of the penalization of the coefficients can be varied to accommodate a large range of regression settings.

Spline-based regression methods are extensively described in the statistical literature. While the theoretical properties of (unpenalized) regression splines and smoothing splines are well established, results for penalized regression splines have only recently become available. The monographs by {{cite|wah90}}, {{cite|eub99}} and {{cite|rup03}} are good sources of information on spline-based methods.

== Acknowledgements ==
Based on an article from Lovric, Miodrag (2011), International
Encyclopedia of Statistical Science. Heidelberg: Springer Science
+Business Media, LLC.


====References====
{|
|-
|valign="top"|{{Ref|eub99}}||valign="top"| Eubank, R. L. (1999).  ''Nonparametric Regression and Spline Smoothing'' (2nd ed.).  New York: Marcel Dekker. 
|-
|valign="top"|{{Ref|fan96}}||valign="top"| Fan, J. and I. Gijbels (1996).  ''Local Polynomial Modelling and its Applications''.  London: Chapman & Hall. 
|-
|valign="top"|{{Ref|rup03}}||valign="top"| Ruppert, D., M. P. Wand, and R. J. Carroll (2003).  ''Semiparametric Regression''.  Cambridge, UK: Cambridge University Press. 
|-
|valign="top"|{{Ref|wah90}}||valign="top"| Wahba, G. (1990). ''Spline models for observational data''. SIAM [Society for Industrial and Applied Mathematics].
|-
|valign="top"|{{Ref|wan95}}||valign="top"| Wand, M. P. and M. C. Jones (1995).  ''Kernel Smoothing''.  London: Chapman and Hall.
|-
|}

<!-- \end{document} -->

<references />

[[Category:Statprob]]