2
$\begingroup$

Let $(\mathcal{H}, \mathcal{K})$ be a reproducing kernel Hilbert space and denote $\mathcal{K}_x := \mathcal{K}(x, \cdot)$.
Is there a simple way to prove $f_n \to_\mathcal{H} f$ (shorthand for $\|f_n - f\|_\mathcal{H} \to 0$) implies $f_n \to f$ pointwise without having to invoke the reproducing property $\langle g, \mathcal{K}_x \rangle_\mathcal{H} = g(x)$?

I'm likely misunderstanding something, but it appears to me that this result is needed to prove the reproducing property itself for functions not in $\mathcal{S} := \text{span}\{\mathcal{K}_x | x \in \mathcal{X}\}$ but in its completion when constructing the RKHS $\mathcal{H}$.
It is easy to see that any function in $\mathcal{S}$ satisfies the reproducing property.
For $f \in \mathcal{H} \setminus \mathcal{S}$, by definition we can always find a sequence $f_n$ in $\mathcal{S}$ such that $f_n \to_\mathcal{H} f$. Then,

\begin{align*} \langle f, \mathcal{K}_x \rangle_\mathcal{H} &= \lim_{n \to \infty} \langle f_n, \mathcal{K}_x \rangle_\mathcal{H} \quad \text{since } \langle \cdot, \mathcal{K}_x \rangle_\mathcal{H} \text{ is continuous w.r.t } \| \cdot \|_\mathcal{H} \\ &= \lim_{n \to \infty} f_n(x) \quad \text{by the reproducing property for functions in } \mathcal{S} \\ &= f(x) \quad \text{assuming we can use the fact that } f_n \to_\mathcal{H} f \text{ implies } f_n \to f. \end{align*}

Edit:
To be clear, I'm following the proof of Theorem 12.11 from Wainwright's High-dimensional statistics, which states that

For any positive semidefinite kernel $\mathcal{K}$, there is a unique Hilbert space $\mathcal{H} \subset \mathbb{R}^\mathcal{X}$ in which the kernel satisfies the reproducing property: $$\langle f, \mathcal{K}_x \rangle_\mathcal{H} = f(x) \quad \forall f \in \mathcal{H}.$$ $\mathcal{H}$ is known as the RKHS associated with $\mathcal{K}$.

Therefore, instead of defining the RKHS to be a space of bounded evaluation functionals (this appears as a theorem in the book), the goal is to construct the RKHS by completing the span of $\{\mathcal{K}_x\}_{x \in \mathcal{X}}$ and show that it satisfies all the required properties.

$\endgroup$
4
  • $\begingroup$ Isn't this guaranteed by the definition of RKHS? i.e., the functional $f \mapsto f(x)$ is linear and continous. $\endgroup$
    – daw
    Nov 30, 2023 at 9:28
  • $\begingroup$ @daw Thanks for the comment! Can you clarify what this entails? I know this can be used in conjunction with the reproducing property to prove that RKHS convergence implies pointwise convergence, but this creates a tautology if I then want to apply this result to prove the reproducing property. $\endgroup$
    – Sleeper
    Nov 30, 2023 at 23:20
  • $\begingroup$ The definition of RKHS on wikipedia is that the evaluation functionals $f\mapsto f(x)$ are continuous wrt to the RKHS-norm. Then $f_n \to f$ in $H$ implies $f_n(x)\to f(x)$ for all $x$. $\endgroup$
    – daw
    Dec 1, 2023 at 7:04
  • $\begingroup$ @daw Ah I realize where the confusion is - I'm starting from a different definition of RKHS. Instead of defining it to be the space of continuous/bounded evaluation functionals, the book I'm following defines it to be the unique Hilbert space that satisfies the reproducing property. I'll update the post to be more clear. $\endgroup$
    – Sleeper
    Dec 1, 2023 at 23:47

1 Answer 1

0
$\begingroup$

As mentioned by @daw, for a Hilbert space $\mathcal{H}$ to become a RKHS the evaluation functionals $L_{x}​(f):=f(x)$ must be continuous, which is equivalent to being bounded. Thus for every $x$ there is a $M_{x}$ with $L_{x}(f)=f(x)\leq M_{x} \| f \|_{\mathcal{H}}$.

It follows that $f_{n} \rightarrow f$ implies the pointwise convergence. Pick an arbitrary $x$ then $$ |f(x)-f_{n}(x)|=|(f-f_{n})(x)|=|L_{x}(f-f_{n})| \leq M_{x}\|f-f_{n}\| $$ and since $\|f_{n} - f\| \rightarrow 0$ we see that $f_{n}(x) \rightarrow f(x)$.

$\endgroup$
10
  • $\begingroup$ Thanks for the answer! I know of this proof but wasn't able to apply it since the book I'm following starts from a different definition (see edit). The second equality of your equation is where I got stumped - when written as an inner product it is the reproducing property, which is exactly what I was trying to prove. $\endgroup$
    – Sleeper
    Dec 2, 2023 at 0:02
  • $\begingroup$ I think in some way or another you will need to use this property since this is what distinguishes a RKHS from a Hilbert space. And in a Hilbert space your implication does not hold as can be seen f.e. by considering $L^{2}$. $\endgroup$
    – bayes2021
    Dec 2, 2023 at 16:18
  • $\begingroup$ What I gathered from the book is that the property should arise naturally from a given positive semidefinite kernel. This video showcases a similar proof (theorem at 4:00): y2u.be/EoM_DF3VAO8 However, it also skimmed over the reproducing property part for functions in the completion (28:54). $\endgroup$
    – Sleeper
    Dec 3, 2023 at 22:31
  • 1
    $\begingroup$ I think what you are looking for is he Moore-Aronszajn theorem. As you can see, the reproducing property arises from the definition of the inner product on $\mathcal{S}$. $\endgroup$
    – bayes2021
    Dec 4, 2023 at 15:25
  • $\begingroup$ Yes this is exactly the theorem I'm looking for. My original question boils down to justifying taking the infinite sum out of the inner product, which I assume will be a result of some convergence theorem (getting too rusty on these...) Can you update your answer so I can accept it? Thanks! $\endgroup$
    – Sleeper
    Dec 6, 2023 at 3:55

You must log in to answer this question.