View source for Empirical process

A [[Stochastic process|stochastic process]] constructed from a [[Sample|sample]] and the corresponding [[Probability measure|probability measure]]. Let <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e1100801.png" /> be a sequence of independent random elements with common law <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e1100802.png" />, taking values in a [[Measurable space|measurable space]] <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e1100803.png" />. The empirical measure <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e1100804.png" /> of the first <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e1100805.png" /> <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e1100806.png" />s is the discrete random measure that places mass <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e1100807.png" /> on each such <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e1100808.png" />:

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e1100809.png" /></td> </tr></table>

Obviously, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008010.png" /> is binomially distributed with parameters <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008011.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008012.png" /> (cf. [[Binomial distribution|Binomial distribution]]). Hence <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008013.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008014.png" />, and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008015.png" /> converges in distribution, as <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008016.png" />, to a centred normal random variable with variance <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008017.png" /> (cf. [[Convergence in distribution|Convergence in distribution]]). Therefore it is natural to define an empirical process indexed by sets by

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008018.png" /></td> <td valign="top" style="width:5%;text-align:right;">(a1)</td></tr></table>

where <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008019.png" />. If <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008020.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008021.png" />, one writes <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008022.png" /> for the empirical distribution function, and the empirical process specializes to the classical empirical process

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008023.png" /></td> <td valign="top" style="width:5%;text-align:right;">(a2)</td></tr></table>

where <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008024.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008025.png" />, is the distribution function of the elements <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008026.png" />. Replacing sets by their indicator functions leads, more generally, to the definition of an empirical process indexed by functions:

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008027.png" /></td> <td valign="top" style="width:5%;text-align:right;">(a3)</td></tr></table>

where

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008028.png" /></td> </tr></table>

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008029.png" /></td> </tr></table>

and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008030.png" /> is a suitable class of measurable functions from <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008031.png" /> to <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008032.png" />.

The main aim of the theory of empirical processes is to obtain results uniformly in <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008033.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008034.png" /> or <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008035.png" />; in particular, Glivenko–Cantelli-type theorems, central limit theorems, laws of the iterated logarithm, and probability inequalities (cf., e.g., [[Empirical distribution|Empirical distribution]]; [[Central limit theorem|Central limit theorem]]; [[Law of the iterated logarithm|Law of the iterated logarithm]]). (Measurability issues will be disregarded in the sequel.) The concept of a [[Vapnik–Chervonenkis class|Vapnik–Chervonenkis class]] plays an important role in set-indexed situations. E.g., if <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008036.png" /> is a Vapnik–Chervonenkis class, then for every probability measure <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008037.png" /> on <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008038.png" />,

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008039.png" /></td> <td valign="top" style="width:5%;text-align:right;">(a4)</td></tr></table>

and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008040.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008041.png" />, converges weakly (see [[#References|[a10]]] and [[Weak topology|Weak topology]]) to <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008042.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008043.png" />, a centred, bounded [[Gaussian process|Gaussian process]], which is uniformly continuous (with respect to the pseudometric <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008044.png" /> defined by <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008045.png" />) and has covariance structure

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008046.png" /></td> </tr></table>

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008047.png" /></td> </tr></table>

For the classical empirical process in (a2), this limiting process specializes to <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008048.png" />, where <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008049.png" /> is a Brownian bridge (cf. [[Non-parametric methods in statistics|Non-parametric methods in statistics]]). A sharp version of the first result is the following: (a4) holds if and only if

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008050.png" /></td> <td valign="top" style="width:5%;text-align:right;">(a5)</td></tr></table>

where

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008051.png" /></td> </tr></table>

(see [[Vapnik–Chervonenkis class|Vapnik–Chervonenkis class]]). A corresponding sharp version of the [[Central limit theorem|central limit theorem]] exists too; essentially the only change is that the <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008052.png" /> in the denominator of (a5) has to be replaced by <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008053.png" /> to obtain an  "if and only if"  condition for the central limit theorem. Other useful concepts in connection with empirical processes are various notions of [[Entropy|entropy]], see [[#References|[a12]]], [[#References|[a13]]], [[#References|[a9]]], [[#References|[a10]]]. Also, for the function-indexed process in (a3), the analogues of (a4) and the central limit theorem above have been studied thoroughly, see [[#References|[a5]]], [[#References|[a9]]], [[#References|[a10]]].

For the classical empirical process in (a2), approximation theorems which yield a rate of convergence in the central limit theorem are extremely useful: A sequence of Brownian bridges <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008054.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008055.png" />, can be constructed such that for all <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008056.png" />

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008057.png" /></td> </tr></table>

A similar, only slightly less sharp, result can be obtained for the situation where the joint distribution of the <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008058.png" />s is known, i.e., the <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/e/e110/e110080/e11008059.png" />s are defined by means of one single Kiefer process, see [[#References|[a3]]].

Empirical and related processes have many applications in many different subfields of probability theory and (non-parametric) statistics.

====References====
<table><TR><TD valign="top">[a1]</TD> <TD valign="top">  K.S. Alexander,   "Rates of growth and sample moduli for weighted empirical processes indexed by sets"  ''Probab. Th. Rel. Fields'' , '''75'''  (1987)  pp. 379–423</TD></TR><TR><TD valign="top">[a2]</TD> <TD valign="top">  M. Csörgő,   S. Csörgő,   L. Horváth,   D.M. Mason,   "Weighted empirical and quantile processes"  ''Ann. of Probab.'' , '''14'''  (1986)  pp. 31–85</TD></TR><TR><TD valign="top">[a3]</TD> <TD valign="top">  M. Csörgő,   P. Révész,   "Strong approximations in probability and statistics" , Acad. Press  (1981)</TD></TR><TR><TD valign="top">[a4]</TD> <TD valign="top">  P. Deheuvels,   D.M. Mason,   "Functional laws of the iterated logarithm for the increments of empirical and quantile processes"  ''Ann. of Probab.'' , '''20'''  (1992)  pp. 1248–1287</TD></TR><TR><TD valign="top">[a5]</TD> <TD valign="top">  R.M. Dudley,   "Universal Donsker classes and metric entropy"  ''Ann. of Probab.'' , '''15'''  (1987)  pp. 1306–1326</TD></TR><TR><TD valign="top">[a6]</TD> <TD valign="top">  J.H.J. Einmahl,   "The a.s. behavior of the weighted empirical process and the LIL for the weighted tail empirical process"  ''Ann. of Probab.'' , '''20'''  (1992)  pp. 681–695</TD></TR><TR><TD valign="top">[a7]</TD> <TD valign="top">  E. Giné,   "Empirical processes and applications: an overview"  ''Bernoulli'' , '''2'''  (1996)  pp. 1–28</TD></TR><TR><TD valign="top">[a8]</TD> <TD valign="top">  P. Massart,   "The tight constant in the Dvoretzky–Kiefer–Wolfowitz inequality"  ''Ann. of Probab.'' , '''18'''  (1990)  pp. 1269–1283</TD></TR><TR><TD valign="top">[a9]</TD> <TD valign="top">  D. Pollard,   "Convergence of stochastic processes" , Springer  (1984)</TD></TR><TR><TD valign="top">[a10]</TD> <TD valign="top">  A. Sheehy,   J.A. Wellner,   "Uniform Donsker classes of functions"  ''Ann. of Probab.'' , '''20'''  (1992)  pp. 1983–2030</TD></TR><TR><TD valign="top">[a11]</TD> <TD valign="top">  G.R. Shorack,   J.A. Wellner,   "Empirical processes with applications to statistics" , Wiley  (1986)</TD></TR><TR><TD valign="top">[a12]</TD> <TD valign="top">  K.S. Alexander,   "Probability inequalities for empirical processes and a law of the iterated logarithm"  ''Ann. of Probab.'' , '''12'''  (1984)  pp. 1041–1067</TD></TR><TR><TD valign="top">[a13]</TD> <TD valign="top">  K.S. Alexander,   "Correction: Probability inequalities for empirical processes and a law of the iterated logarithm"  ''Ann. of Probab.'' , '''15'''  (1987)  pp. 428–430</TD></TR></table>