3
$\begingroup$

Let $K: X \times X \rightarrow \mathbb{C}$ be a positive definite kernel on a set $X$, i.e. for any $x_1, \cdots, x_n \in X$, the matrix

$$ [K(x_i, x_j)]_{ij} \in \mathbb{C}^{n \times n} $$

is positive definite.

The reproducing kernel Hilbert space $\mathcal{H}_K$ with kernel $K$ is the Hilbert space closure of the set $\{ K_x(y) = K(x,y)\}$ with the inner product

$$ \langle K_x(y), K_{x'}(y) \rangle_{\mathcal{H}_K} = K(x, x'). $$

Question

Consider the vector space of functions on $X$ (no additional structure is assumed on $X$). Define an inner product by

$$ \langle \delta_x, \delta_{x'} \rangle = K(x, x'). $$

Then the resulting Hilbert space $\mathcal{H}$ is isomorphic to $\mathcal{H}_{K}$ via the isomorphism

$$ \delta_x(y) \mapsto K_x(y). $$

  1. $\mathcal{H}$ seems a much more natural and straight forward object than $\mathcal{H}_K$, so what is gained by the "reproducing kernel" construction? The reproducing kernel construction makes pointwise evaluation an element of the dual...why is this useful?

  2. In particular, what are some examples of specific contexts where $\mathcal{H}_K$ is more handy than $\mathcal{H}$ (I am guessing they must exist)?

$\endgroup$
2
$\begingroup$

A very practical application of this is to Machine Learning, specifically Support Vector Machines and general kernel methods. The problem setup is that you have a bunch of points $(\mathbf{x}_i,y_i)$, with $y_i$ being the value you want to predict from $\mathbf{x}_i$. So you start with some training set of points, and you want to construct a function that predicts $y$ given $\mathbf{x}$. If you want a linear predictor, then usual optimization theory works. But if you want a non-linear predictor, then things are much more difficult. Specifically, what happens is that it becomes extremely expensive to do operations as simple as multiplication and addition if your space $X$ is huge, say 50 dimensional. As well, with something like a Support Vector Machine, you are trying to effectively find a transformation $\phi$ of your set of points so that you can split different categories of points by hyperplanes. You can imagine how difficult such a problem would be directly.

The brilliance of kernel methods, specifically involving the so called kernel trick is that one can reduce these kinds of problems to calculating inner products of your data (which is relatively fast), and then working with the reproducing kernel, where one has access to Mercer's theorem.

$\endgroup$
  • $\begingroup$ So if $\mathcal{H}$ and $\mathcal{H}_K$ are isomorphic, why can the same not be done with just $\mathcal{H}$? $\endgroup$ – Michael Nov 27 '14 at 6:28
  • $\begingroup$ More precisely, what does that specific representation via reproducing kernel give you in that context? $\endgroup$ – Michael Nov 28 '14 at 7:27
  • $\begingroup$ The main use is to go from a infinite dimensional space $\mathcal{H}$ to a finite dimensional one. $\endgroup$ – Næreen Feb 10 '16 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.