# Reproducing kernel Hilbert space, why?

Let $K: X \times X \rightarrow \mathbb{C}$ be a positive definite kernel on a set $X$, i.e. for any $x_1, \cdots, x_n \in X$, the matrix

$$[K(x_i, x_j)]_{ij} \in \mathbb{C}^{n \times n}$$

is positive definite.

The reproducing kernel Hilbert space $\mathcal{H}_K$ with kernel $K$ is the Hilbert space closure of the set $\{ K_x(y) = K(x,y)\}$ with the inner product

$$\langle K_x(y), K_{x'}(y) \rangle_{\mathcal{H}_K} = K(x, x').$$

Question

Consider the vector space of functions on $X$ (no additional structure is assumed on $X$). Define an inner product by

$$\langle \delta_x, \delta_{x'} \rangle = K(x, x').$$

Then the resulting Hilbert space $\mathcal{H}$ is isomorphic to $\mathcal{H}_{K}$ via the isomorphism

$$\delta_x(y) \mapsto K_x(y).$$

1. $\mathcal{H}$ seems a much more natural and straight forward object than $\mathcal{H}_K$, so what is gained by the "reproducing kernel" construction? The reproducing kernel construction makes pointwise evaluation an element of the dual...why is this useful?

2. In particular, what are some examples of specific contexts where $\mathcal{H}_K$ is more handy than $\mathcal{H}$ (I am guessing they must exist)?

A very practical application of this is to Machine Learning, specifically Support Vector Machines and general kernel methods. The problem setup is that you have a bunch of points $(\mathbf{x}_i,y_i)$, with $y_i$ being the value you want to predict from $\mathbf{x}_i$. So you start with some training set of points, and you want to construct a function that predicts $y$ given $\mathbf{x}$. If you want a linear predictor, then usual optimization theory works. But if you want a non-linear predictor, then things are much more difficult. Specifically, what happens is that it becomes extremely expensive to do operations as simple as multiplication and addition if your space $X$ is huge, say 50 dimensional. As well, with something like a Support Vector Machine, you are trying to effectively find a transformation $\phi$ of your set of points so that you can split different categories of points by hyperplanes. You can imagine how difficult such a problem would be directly.
• So if $\mathcal{H}$ and $\mathcal{H}_K$ are isomorphic, why can the same not be done with just $\mathcal{H}$? – Michael Nov 27 '14 at 6:28
• The main use is to go from a infinite dimensional space $\mathcal{H}$ to a finite dimensional one. – Næreen Feb 10 '16 at 10:27