Reproducing-kernel Hilbert space

Let $H$ be a Hilbert space of functions defined on an abstract set $E$.

Let $( f , g )$ denote the inner product and let $\| f \| = ( f , f ) ^ { 1 / 2 }$ be the norm in $H$. The space $H$ is called a reproducing-kernel Hilbert space if there exists a function $K ( x , y )$, the reproducing kernel, on $E \times E$ such that:

1) $K ( x , y ) \in H$ for any $y \in E$;

2) $( f ( . ) , K (. , y ) ) = f ( y )$ for all $f \in H$ (the reproducing property). From this definition it follows that the value $f ( y )$ at a point $y \in E$ is a continuous linear functional in $H$:

\begin{equation*} | f ( y ) | \leq c ( y ) \| f \| , c ( y ) : = \| K (\, .\, , y ) \|. \end{equation*}

The converse is also true. The following theorem holds: A Hilbert space of functions on a set $E$ is a reproducing-kernel Hilbert space if and only if $| f ( y ) | \leq c ( y ) \| f \|$ for all $y \in E$.

By the Riesz theorem, the above assumption implies the existence of a linear functional $K (\ \cdot \ , y )$ such that $f ( y ) = ( f , K (\, .\, , y ) )$. By the construction, the kernel $K ( x , y )$ is the reproducing kernel for $H$.

An example of a construction of a reproducing-kernel Hilbert space is the rigged triple of Hilbert spaces $H _ { + } \subset H _ { 0 } \subset H _ { - }$, which is defined as follows [a5] (cf. also Rigged Hilbert space). Let $H _ { 0 }$ be a Hilbert space of functions, let $A > 0$ be a linear densely defined self-adjoint operator on $H _ { 0 }$, $A \varphi _ { j } = \lambda _ { j } \varphi _ { j }$ (the eigenvalues $\lambda_j > 0$ are counted according to their multiplicities) and assume that

\begin{equation*} \Lambda ^ { 2 } : = \sum _ { j = 1 } ^ { \infty } \lambda _ { j } < \infty , | \varphi _ { j } ( x ) | < c , \forall j , x. \end{equation*}

Define $H _ { - } \supset H _ { 0 }$ to be the Hilbert space with inner product $( u , v ) _ { - } = ( A ^ { 1 / 2 } u , A ^ { 1 / 2 } v ) _ { 0 }$. $H_-$ is the completion of $H _ { 0 }$ in the norm $\| u \| : = ( u , u ) ^ { 1 / 2 }_ { - }$. Let $H _ { + } \subset H _ { 0 }$ be the dual space to $H_-$ with respect to $H _ { 0 }$. Then the inner product in $H _ { + }$ is defined by the formula $( u , v ) _ { + } : = ( A ^ { - 1 / 2 } u , A ^ { - 1 / 2 } v ) _ { 0 },$ and $H _ { + } = R ( A ^ { 1 / 2 } )$, equipped with the inner product $( u , v )_+$, is a Hilbert space.

Define $B ( x , y ) = \sum _ { j = 1 } ^ { \infty } \lambda _ { j } \overline { \varphi _ { j } ( x ) } \varphi _ { j } ( y )$, where the overline stands for complex conjugation. For any $y$, one has $B ( x , y ) \in H _ { + }$. Indeed,

\begin{equation*} \| B ( x , y ) \| _ { + } \leq c \sum _ { j = 1 } ^ { \infty } \| \lambda _j \varphi_j ( x ) \| _ { + } = \end{equation*}

\begin{equation*} = c \sum _ { j = 1 } ^ { \infty } ( A \varphi _ { j } , \varphi _ { j } ) _ { 0 } = c \Lambda ^ { 2 } < \infty. \end{equation*}

Furthermore,

\begin{equation*} ( u , B ( x , y ) ) _ { + } = ( u , A ^ { - 1 } B ) = u ( y ), \end{equation*}

so that $B ( x , y )$ is the reproducing kernel in $H _ { + }$. Moreover $| u ( y ) | \leq c ( y ) \| u \|_+$, where $c ( y ) > 0$ is a constant independent of $u \in H _ { + }$. Indeed, if $u _ { j } : = ( u , \varphi _ { j } ) _ { 0 }$ and $u \in H _ { + }$, then $u = A ^ { 1 / 2 } v$, $v \in H _ { 0 }$ $v _ { j } \lambda _ { j } ^ { 1 / 2 } = u _ { j }$, and $| u ( y ) | \leq \sum _ { j = 1 } ^ { \infty } | u _ { j } \varphi _ { j } ( y ) | \leq c \Lambda \| v \| = c \Lambda \| u \| _ { + }$.

Thus $H _ { + }$ is a reproducing kernel Hilbert space with the reproducing kernel $B ( x , y )$ defined above. If $K ( x , y )$ is a function on $E \times E$ such that

\begin{equation} \tag{a1} \sum _ { i ,\, j = 1 } ^ { n } K ( x _ { i } , x _ { j } ) t _ { j } \overline { t } _ { i } \geq 0 ,\, \forall t \in \mathbf{C} ^ { n } ,\, \forall x _ { i } \in E, \end{equation}

then one can define a pre-Hilbert space $H ^ { 0 }$ of functions of the form

\begin{equation*} f ( x ) : = \sum _ { j = 1 } ^ { J } K ( x , y_j ) c_j , c_j =\text{const.} \end{equation*}

The inner product of two functions from $H ^ { 0 }$ is defined by

\begin{equation*} ( f , g ) : = \left( \sum _ { j = 1 } ^ { J } K ( x , y _ { j } ) c _ { j } , \sum _ { m = 1 } ^ { M } K ( x , z _ { m } ) \beta _ { m } \right) = \end{equation*}

\begin{equation*} = \sum _ { j , m } K ( z _ { m } , y _ { j } ) c _ { j } \overline { \beta _ { m } }. \end{equation*}

This definition makes sense because of (a1) and because of reproducing property 2). In particular, $( f , f ) \geq 0$, as follows from (a1), and if $( f , f ) = 0$ then $f = 0$, as follows from property 2).

Indeed,

\begin{equation*} ( f ( x ) , K ( x , y ) ) = \left( \sum _ { j = 1 } ^ { J } K ( x , y _ { j } ) c _ { j } , K ( x , y ) \right) = \end{equation*}

\begin{equation*} = \sum _ { j = 1 } ^ { J } K ( y , y _ { j } ) c _ { j } = f ( y ) , \forall y \in E. \end{equation*}

Thus, if $( f , f ) = 0$, then $\| f \| = 0$ and $| f ( y ) | \leq \| f \| \| K ( x , y ) \| = 0$, so $f ( y ) = 0$ as claimed.

Denote by $H$ the completion of $H ^ { 0 }$ in the norm $\| f \|$. Then $H$ is a reproducing-kernel Hilbert space and $K ( x , y )$ is its reproducing kernel.

A reproducing-kernel Hilbert space is uniquely defined by its reproducing kernel. Indeed, if $H _ { 1 }$ is another reproducing-kernel Hilbert space with the same reproducing kernel $K ( x , y )$, then $H ^ { 0 } \subset H _ { 1 }$ and $H ^ { 0 }$ is dense in $H _ { 1 }$: If $f \in H _ { 1 }$, $0 = ( f , K ( x , y ) ) _ { H _ { 1 } } = f ( y )$ for all $y \in E$, then $f \equiv 0$. Using this and the equality $( f , g ) _ { H _ { 1 } } = ( f , g ) _ { H }$ for all $f , g \in H ^ { 0 }$, one can check that $H \subset H _ { 1 }$ and vice versa, so $H = H _ { 1 }$, that is, $H$ and $H _ { 1 }$ consist of the same set of elements. Moreover, the norms in $H$ and $H _ { 1 }$ are equal. Indeed, take an arbitrary $f \in H _ { 1 }$ and a sequence $f _ { n } \in H ^ { 0 }$, $\| f _ { n } - f \| _ { 1 } \rightarrow 0$. Then

\begin{equation*} \| f \| _ { 1 } ^ { 2 } = \operatorname { lim } _ { n \rightarrow \infty } \| f _ { n } \| _ { 1 } ^ { 2 } = \end{equation*}

\begin{equation*} = \operatorname { lim } _ { n \rightarrow 0 } \left( \sum _ { j_n = 1 } ^ { J _ { n } } K ( x , y _ { j _n } ) c _ { j _n } , \sum _ { m_n = 1 } ^ { J _ { n } } K ( x , y _ { m_n } ) c _ { m_n } \right) _ { 1 } = \end{equation*}

\begin{equation*} = \sum _ { j _ { n } ,\, m _ { n } } ^ { J _ { n } } K ( y _ { m _ { n } } , y _ { j _ { n } } ) c _ { j _ { n } } \overline { c_{m _ { n }}} = \end{equation*}

\begin{equation*} = \operatorname { lim } _ { n \rightarrow \infty } ( f _ { n } , f _ { n } ) = \| f \| ^ { 2 }. \end{equation*}

Thus, the norms in $H _ { 1 }$ and $H$ are equal, as claimed, and so are the inner products (by the polarization identity).

Define a linear operator $L : {\cal H} \rightarrow H$, $D ( L ) = \mathcal{H}$, where $\mathcal{H} = L ^ { 2 } ( T , d m )$ and $H$ is the range $R ( L )$ of $L$, which will be equipped with the structure of a Hilbert space below:

\begin{equation} \tag{a2} f ( x ) = L F : = \int _ { T } F ( t ) \overline { h ( t , x ) } d m ( t ). \end{equation}

Here, $T$ is a domain in ${\bf R} ^ { n }$ and $m$ is a positive measure on $T$, $m ( T ) < \infty$, $h ( t , x ) \in \mathcal{H}$ for all $x \in E$, and it is assumed that $L$ is injective, that is, the system $\{ h ( t , x ) \}_{\forall x \in E}$ is total in $\mathcal{H}$ (cf. also Total set).

Define

\begin{equation} \tag{a3} K ( x , y ) : = \int _ { T } h ( t , y ) \overline { h ( t , x ) } d m ( t ) = \end{equation}

\begin{equation*} = ( h (\, . \, , y ) , h (\, . \, , x ) ) _ { \mathcal{H} }. \end{equation*}

This kernel clearly satisfies condition (a1) and therefore is a reproducing kernel for the reproducing-kernel Hilbert space $H _ { K }$ which it generates. Clearly $K ( x , y ) \in H$ for all $y \in E$. If $f \in H$, that is, $f = L F$, $f \in \mathcal{H}$, then

\begin{equation*} ( f ( \cdot ) , K ( \cdot , y ) ) _ { H } = ( L F , K ( \cdot , y ) ) _ { H } = \end{equation*}

\begin{equation*} = ( ( F ( \cdot ) , h ( \cdot , x ) ) _ { \mathcal{H} } , ( h ( \text{..} , y ) , h ( \text{..} , x ) ) _ { \mathcal{H} } ) _ { H } = \end{equation*}

\begin{equation*} = ( F ( . ) , ( h ( .. , y ) , ( h (. , x ) , h ( .. , x ) ) _ { H } ) _ {\cal H } ) _ {\cal H } = \end{equation*}

\begin{equation*} = ( F ( \cdot ) , h ( \cdot , y ) ) _ { \mathcal{H} } = f ( y ), \end{equation*}

if one equips $H$ with the inner product such that $( f , g ) _ { H } = ( F , G ) _ { \mathcal{H} }$. This requirement is formally equivalent to the following one: $( h ( s , x ) , h ( t , x ) ) _ { H } = \delta _ { m } ( t - s )$, where $( h ( s , y ) , \delta _ { m } ( t - s ) ) _ { \mathcal{H} } = h ( t , y )$, so that the distributional kernel $\delta _ { m } ( t - s )$ is not the usual delta-function, but the one which acts by the rule

\begin{equation*} \int _ { T } d m ( t ) F ( t ) \int _ { T } d m ( s ) G ( s ) \delta _ { m } ( t - s ) = \end{equation*}

\begin{equation*} = \int _ { T } d m ( t ) F ( t ) G ( t ), \end{equation*}

and formally one has $\int _ { T } d m ( s ) G ( s ) \delta _ { m } ( t - s ) = G ( t )$.

With the inner product $( f , g ) _ { H }$, the linear set $R ( L )$ becomes a Hilbert space:

\begin{equation} \tag{a4} ( f , g ) _ { H } = ( L F , L G ) _ { H } = \end{equation}

\begin{equation*} = \int_{T} \int _ { T } d m ( t ) d m ( s ) F ( t ) \overline { G ( s ) } ( h ( s , x ) , h ( t , x ) ) _ { H } = \end{equation*}

\begin{equation*} = \int _ { T } d m ( t ) F ( t ) \overline { G ( t ) } = ( F , G ) _ { \mathcal{H} }. \end{equation*}

Thus, this inner product makes $L$ an isometric operator defined on all of $\mathcal{H}$ and makes $H = R ( L )$ a (complete) Hilbert space, namely $H = H _ { K }$, a reproducing-kernel Hilbert space. Since $L$ is assumed injective, it follows that $L ^ { - 1 }$ is defined on all of $R ( L ) = H$ and, since $H$ is complete in the norm $\|\, f \| = ( f , f ) ^ { 1 / 2 } _ { H }$, one concludes that $L ^ { - 1 }$ is continuous (by the Banach theorem). Consequently, $L$ is a co-isometry, that is, $L ^ { * } = L ^ { - 1 }$, where $L ^ { * }$ is the adjoint operator to $L$. If $L ^ { * } = L ^ { - 1 }$, then one can write an inversion formula for the linear transform $L$ similar to the well-known inversion formula for the Fourier transform. Formally one has:

\begin{equation*} f ( x ) = ( F ( t ) , h ( t , x ) ) _ { \cal H } , ( f ( x ) , h ( s , x ) ) _ { H } = F ( s ). \end{equation*}

The space $H = H _ { K }$ is the reproducing-kernel Hilbert space generated by kernel (a3) which is the reproducing kernel for $H$. The above formal inversion formulas may be of practical interest if the norm in $H$ is a standard one. In this case the second formula should be suitably interpreted, since $F ( s )$ is defined at $m$-almost all $s$.

In [a6] it is claimed that the characterization of the range of the linear operator $L$, defined in (a3), can be given as follows: $R ( L ) = H _ { K }$, where $H _ { K }$ is the reproducing-kernel Hilbert space generated by kernel (a3).

However, in fact such a characterization does not give, in general, practically useful necessary and sufficient conditions for $f ( x ) \in R ( L )$ because the norm in $H _ { K }$ is not defined in terms of standard norms such as Sobolev or Hölder ones (see [a3], [a4], [a5]). However, when the norm in $H _ { K }$ is equivalent to a standard norm, the above characterization becomes efficient (see [a3], [a4], [a5], and also [a6]).

Many concrete examples of reproducing-kernel Hilbert spaces can be found in [a1], [a2] and [a6].

The papers [a1] and [a7] are important in this area, the book [a6] contains many references, while [a2] is an earlier book important for the development of the theory of reproducing-kernel Hilbert spaces.

References

[a1]	N. Aronszajn, "Theory of reproducing kernels" Trans. Amer. Math. Soc. , 68 (1950) pp. 337–404
[a2]	S. Bergman, "The kernel function and conformal mapping" , Amer. Math. Soc. (1950)
[a3]	A.G. Ramm, "On the theory of reproducing kernel Hilbert spaces" J. Inverse Ill-Posed Probl. , 6 : 5 (1998) pp. 515–520
[a4]	A.G. Ramm, "On Saitoh's characterization of the range of linear transforms" A.G. Ramm (ed.) , Inverse Problems, Tomography and Image Processing , Plenum (1998) pp. 125–128
[a5]	A.G. Ramm, "Random fields estimation theory" , Longman/Wiley (1990)
[a6]	S. Saitoh, "Integral transforms, reproducing kernels and their applications" , Pitman Res. Notes , Longman (1997)
[a7]	L. Schwartz, "Sous-espaces hilbertiens d'espaces vectoriels topologiques et noyaux associès (noyaux reproduisants)" J. Anal. Math. , 13 (1964) pp. 115–256