
In
functional analysis
Functional analysis is a branch of mathematical analysis, the core of which is formed by the study of vector spaces endowed with some kind of limit-related structure (for example, Inner product space#Definition, inner product, Norm (mathematics ...
, a reproducing kernel Hilbert space (RKHS) is a
Hilbert space of functions in which point evaluation is a continuous
linear functional. Specifically, a Hilbert space
of functions from a set
(to
or
) is an RKHS if the point-evaluation functional
,
, is continuous for every
. Equivalently,
is an RKHS if there exists a function
such that, for all
,
The function
is then called the ''reproducing kernel'', and it reproduces the value of
at
via the inner product.
An immediate consequence of this property is that convergence in norm implies uniform convergence on any subset of
on which
is bounded. However, the converse does not necessarily hold. Often the set
carries a topology, and
depends continuously on
, in which case: convergence in norm implies uniform convergence on compact subsets of
.
It is not entirely straightforward to construct natural examples of a Hilbert space which are not an RKHS in a non-trivial fashion. Some examples, however, have been found.
While, formally,
''L''2 spaces are defined as Hilbert spaces of equivalence classes of functions, this definition can trivially be extended to a Hilbert space of functions by choosing a (total) function as a representative for each equivalence class. However, no choice of representatives can make this space an RKHS (
would need to be the non-existent Dirac delta function). However, there are RKHSs in which the norm is an ''L''
2-norm, such as the space of band-limited functions (see the example below).
An RKHS is associated with a kernel that reproduces every function in the space in the sense that for every
in the set on which the functions are defined, "evaluation at
" can be performed by taking an inner product with a function determined by the kernel. Such a ''reproducing kernel'' exists if and only if every evaluation functional is continuous.
The reproducing kernel was first introduced in the 1907 work of
Stanisław Zaremba concerning
boundary value problems for
harmonic and
biharmonic functions.
James Mercer simultaneously examined
functions which satisfy the reproducing property in the theory of
integral equations. The idea of the reproducing kernel remained untouched for nearly twenty years until it appeared in the dissertations of
Gábor Szegő,
Stefan Bergman, and
Salomon Bochner. The subject was eventually systematically developed in the early 1950s by
Nachman Aronszajn and Stefan Bergman.
These spaces have wide applications, including
complex analysis,
harmonic analysis, and
quantum mechanics
Quantum mechanics is the fundamental physical Scientific theory, theory that describes the behavior of matter and of light; its unusual characteristics typically occur at and below the scale of atoms. Reprinted, Addison-Wesley, 1989, It is ...
. Reproducing kernel Hilbert spaces are particularly important in the field of
statistical learning theory
Statistical learning theory is a framework for machine learning drawing from the fields of statistics and functional analysis. Statistical learning theory deals with the statistical inference problem of finding a predictive function based on da ...
because of the celebrated
representer theorem
For computer science, in statistical learning theory, a representer theorem is any of several related results stating that a minimizer f^ of a regularized Empirical risk minimization, empirical risk functional defined over a reproducing kernel Hi ...
which states that every function in an RKHS that minimises an empirical risk functional can be written as a
linear combination of the kernel function evaluated at the training points. This is a practically useful result as it effectively simplifies the
empirical risk minimization
In statistical learning theory, the principle of empirical risk minimization defines a family of learning algorithms based on evaluating performance over a known and fixed dataset. The core idea is based on an application of the law of large num ...
problem from an infinite dimensional to a finite dimensional optimization problem.
For ease of understanding, we provide the framework for real-valued Hilbert spaces. The theory can be easily extended to spaces of complex-valued functions and hence include the many important examples of reproducing kernel Hilbert spaces that are spaces of
analytic functions.
Definition
Let
be an arbitrary
set and
a
Hilbert space of
real-valued functions on
, equipped with pointwise addition and pointwise scalar multiplication. The
evaluation
In common usage, evaluation is a systematic determination and assessment of a subject's merit, worth and significance, using criteria governed by a set of Standardization, standards. It can assist an organization, program, design, project or any o ...
functional over the Hilbert space of functions
is a linear functional that evaluates each function at a point
,
:
We say that ''H'' is a reproducing kernel Hilbert space if, for all
in
,
is
continuous at every
in
or, equivalently, if
is a
bounded operator on
, i.e. there exists some
such that
Although
is assumed for all
, it might still be the case that
.
While property () is the weakest condition that ensures both the existence of an inner product and the evaluation of every function in
at every point in the domain, it does not lend itself to easy application in practice. A more intuitive definition of the RKHS can be obtained by observing that this property guarantees that the evaluation functional can be represented by taking the inner product of
with a function
in
. This function is the so-called reproducing kernel for the Hilbert space
from which the RKHS takes its name. More formally, the
Riesz representation theorem implies that for all
in
there exists a unique element
of
with the reproducing property,
Since
is itself a function defined on
with values in the field
(or
in the case of complex Hilbert spaces) and as
is in
we have that
:
where
is the element in
associated to
.
This allows us to define the reproducing kernel of
as a function
(or
in the complex case) by
:
From this definition it is easy to see that
(or
in the complex case) is both symmetric (resp. conjugate symmetric) and
positive definite, i.e.
:
for every
The Moore–Aronszajn theorem (see below) is a sort of converse to this: if a function
satisfies these conditions then there is a Hilbert space of functions on
for which it is a reproducing kernel.
Examples
The simplest example of a reproducing kernel Hilbert space is the space
where
is a set and
is the
counting measure on
. For
, the reproducing kernel
is the
indicator function of the one point set
.
Nontrivial reproducing kernel Hilbert spaces often involve
analytic functions, as we now illustrate by example. Consider the Hilbert space of
bandlimited continuous functions
. Fix some
cutoff frequency