1
$\begingroup$

If I understand correctly, the Mercer Theorem states that if $R$ is a continuous positive-type function, then there exists a sequence of eigenfunctions $\phi_n(.)$ and corresponding nonnegative eigenvalues $\lambda_n$ associated with $R$ as an operator on $L^2$, such that $$R(s,t)=\sum_{n=1}^\infty \lambda_n\phi_n(s)\phi_n(t)$$

Let $\mathcal{H}$ be the reproducing-kernel Hilbert space (RKHS) corresponding to $R$ (cf Moore-Aroszajn Theorem). I am reading the section on RKHS in Elements of Statistical Learning and Inference by Hastie et al (2017 edition, https://hastie.su.domains/ElemStatLearn/, p.168), and after positing an above expansion for $R$, the authors state that $$\forall f\in\mathcal{H}\hspace{0.6cm} f(x)=\sum_{i=1}^{\infty}c_i\phi_i(x)$$ where $\phi_i$ are the eigenvalues associated with $R$, and $c$ are suitable scalars.

Here's my problem. The latter statement implies that $\mathcal{H}$ has a countable orthonormal basis, which means that $\mathcal{H}$ is separable. However, not every RKHS is separable. Moreover, a complete countable basis is not implied by the Mercer Theorem, since there is no claim that the given sequence of eigenfunctions is complete, and the Hilbert basis formed by the eigenfunctions of the Hilbert-Schmidt operator $R$ need not be countable.

Are Hastie et al tacitly assuming $\mathcal{H}$ is separable? I see no other way around their claim.

$\endgroup$

1 Answer 1

1
$\begingroup$

You are right that in general, there is no reason for a RKHS to be separable. However, if the kernel $K$ is continuous and the set $\mathcal X$ over which it is defined is separable, then the corresponding RKHS $\mathcal H $ will also be separable. Here's a proof sketch :

Recall that $\mathcal H$ is the completion of $H_0 :=\text{span}\lbrace K(x,\cdot)\mid x\in\mathcal X\rbrace$. Hence if we prove that $H_0$ is separable, separability of $\mathcal H$ will follow. If we denote by $\{x_n\}_{n=1}^\infty\subset \mathcal X$ a dense subset of $\mathcal X$, it is enough to check that $$H_1:=\bigcup_{n=1}^\infty \left\{\sum_{i=1}^n \alpha_i K(x_i,\cdot)\mid \alpha_1,\ldots,\alpha_n\in\mathbb Q\right\} $$ Is dense in $H_0$, since it is clearly countable. But it is easy to check that any function in $H_0$ can indeed be approximated by elements of $H_1$ when $K$ is continuous : for $H_0\ni f = \sum_{k=1}^m \beta_kK(y_k,\cdot)$, approximate each $y_k$ by an $x_{n_k}$ such that $\|K(y_k,\cdot)- K(x_{n_k},\cdot)\|$ is sufficiently small, and approximate each $\beta_k$ by sufficiently close rationals $\tilde\beta_k$, and the result will follow (I let you fill in the details).

In Hastie's book, they consider $\mathcal X \equiv \mathbb R^p$, which is clearly separable, and all the kernels they work with are continuous, so in their setting $\mathcal H$ is indeed separable.

$\endgroup$
2
  • $\begingroup$ Thanks for that, and the edit. I'm going to be super-rigorous (oh no!) with my grammar from now on! $\endgroup$
    – demim00nde
    Mar 2, 2023 at 21:30
  • $\begingroup$ That seems pretty complete to me! That also reminded me that a proof exists in Reproducing Kernel Hilbert Spaces in Probability and Statistics, Berlinet & Thomas-Agnan, 2004 (Theorem 15 and corollaries), but yours more compactly addresses the question. I think the omission of this in Hastie et al is what bothers me about the book as a whole: it's comprehensive, but you come away with doubts rather than certainty. Personal view. $\endgroup$
    – demim00nde
    Mar 3, 2023 at 11:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .