I have heard this phrase quite a lot now, that RKHS is dense in the space of bounded continuous functions ($\mathcal L_2$).

For example, this would be true with

$$f(x) = \sum_{i=1}^N \alpha_i K(x_i,x)$$

for some $\alpha_i$, and $K$ the RBF kernel

$$K(x,y) = \text{exp}\left(-\frac{\|x-y\|_2^2}{\sigma}\right)$$

I guess this also means for any $x_i$ as well, or for infinite $N$. But which theorem / proof is this referring to?

Some options:

Mercer's representation theorem: $K(x,y)$ can be written as a linear combination of $\phi(x)\phi(y)$ where $\phi$ are the eigenfunctions of $K$

Riesz representer theorem: If $H$ is an RKHS, there exists a positive definite kernel function $K$ in which all functions in $H$ can be written this way.

Moore–Aronszajn: Any positive definite function $K$ defines an RKHS.

None of these quite seem like what I need though...

Edit: Here is a promising cuprit! The nonparametric representor theorem in

Scholkopf, Herbrich, and Smola: A Generalized Representer Theorem (2001)

The nonparametric version gives the functional class

$$\mathcal F = \left\{f | f(x) = \sum_{i=1}^\infty \beta_i k(x,z_i), \beta_i\in \mathbb R, z_i \in \mathcal X \subseteq \mathbb R^n, \|f\|< \infty \right\}$$

Then any $f\in \mathcal F$ minimizing some risk function with regularization $g(\|f\|)$ (for any monotonic $g$) has a representation

$$f(x) = \sum_{i=1}^n \alpha_i K(x_i,x).$$

However, it seems a nontrivial extension is needed to show that $\mathcal F$ is dense in $\mathcal L_2$.

  • $\begingroup$ Yeah let's consider just RBF, though technically it shouldn't matter, should it? $\endgroup$ – Y. S. Dec 26 '17 at 2:46
  • $\begingroup$ Did you mean $f(x) = \sum_{i=1}^{N}\alpha_i K(x_i,x)?$ $\endgroup$ – Arin Chaudhuri Dec 26 '17 at 2:46
  • $\begingroup$ Yes, that sounds right. I'll modify. $\endgroup$ – Y. S. Dec 26 '17 at 2:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.