$k,k_1$ and $k_2$ are kernels on $\mathcal{X}\times\mathcal{X}$, and $k=k_1+k_2$, then we have the following properties for the RKHS (Reproducing Kernel Hilbert Space) $\mathcal{H}$, $\mathcal{H}_1$ and $\mathcal{H}_2$:

$$\mathcal{H}=\mathcal{H}_1+\mathcal{H}_2=\left\{f_1+f_2:f_1\in \mathcal{H}_1, f_2\in\mathcal{H}_2\right\}(1)$$

and for any $f\in \mathcal{H}$,

$$ ||f||^2_{\mathcal{H}}=\min_{f_1,f_2}\left\{||f_1||^2_{\mathcal{H_1}}+||f_2||^2_{\mathcal{H_2}}:f_1+f_2=f\right\}(2)$$

Could someone help to prove (1) or (2)?


So $\mathcal{H}$ is the RKHS of $k$, and we want to prove that $\mathcal{H}=\mathcal{H}_1+\mathcal{H}_2$ (1) and that the topology of $\mathcal{H}$ is the one defined by (2).

Preliminaries. First of all, let's notice that, for all $x \in \mathcal{X}$, $$k(x,\cdot)=k_1(x,\cdot)+k_2(x,\cdot) \in \mathcal{H}_1+\mathcal{H}_2.$$

Topology of $\mathcal{H}_1+\mathcal{H}_2$. To be able to prove the reproducing property, let's focus on the topology of $\mathcal{H}_1+\mathcal{H}_2$. We denote $E=\mathcal{H}_1 \times \mathcal{H}_2$. This set is a Hilbert space if we equip it with the norm $$\left \| \cdot \right \|_{E}:(f_1,f_2) \mapsto \sqrt{\left \| f_1 \right \|_{\mathcal{H}_1}^2+\left \| f_2 \right \|_{\mathcal{H}_2}^2},$$in fact, $E$ is the orthogonal direct sum of $\mathcal{H}_1$ and $\mathcal{H}_2$.

We want to compare the topologies of $\mathcal{H}_1+\mathcal{H}_2$ and $E$. A direct link between these spaces is the natural surjection $ s: \left| \begin{array}{rcl} E & \longrightarrow &\mathcal{H}_1+\mathcal{H}_2 \\ (f_1,f_2) & \longmapsto & f_1+f_2 \\ \end{array} \right.$. We are going to try to "make $s$ injective" (of course, "vanilla $s$" is not necessarily injective). In order to do so, let's consider $N=s^{-1}(\{0\})$. We will begin by proving that $N$ is a closed subset of $E$:

Let $(f_n,-f_n)$ be a sequence of elements of $N$ converging in $E$ to $(f,g)$. By definition of the $E$-norm, $(f_n)_{n\geq 1}$ converges in $\mathcal{H}_1$ to $f$ and $(-f_n)_{n\geq 1}$ converges in $\mathcal{H}_2$ to $g$. Since convergence in a RKHS implies ponctual convergence, we will have $f=-g$ an therefore $(f,g) \in N$. $N$ is therefore a closed subset of $E$.

Since $N$ is closed, $E$ is equal to the direct sum of $N$ and its orthogonal complement $O$. The restriction $\tilde s$ of $s$ to $O$ will therefore be a bijection.

Now that we have a linear bijection, we can equip $\mathcal{H}_1+\mathcal{H}_2$ with an Hilbertian structure inherited from $E$: with the norm $\left \| \cdot \right \|_{\mathcal{H}_1+\mathcal{H}_2}:f \mapsto \left \| \tilde{s}^{-1}(f) \right \|_{E}$, $\mathcal{H}_1+\mathcal{H}_2$ is indeed a Hilbert space.

Now we can prove (2). For every couple of functions $c=(f_1,f_2) \in E$, we can use the orthogonal decomposition $E=N \oplus O $ to write $c=(f_1^N,f_2^N)+\tilde{s}^{-1}(f_1+f_2)$ with $(f_1^N,f_2^N) \in N$. Therefore $$\left \| c \right \|_{E}^2 =\left \| (f_1^N,f_2^N) \right \|_{E}^2 + \left \| f_1+f_2 \right \|_{\mathcal{H}_1+\mathcal{H}_2}^2.$$ Therefore, for every function $f=f_1+f_2$ of $\mathcal{H}_1+\mathcal{H}_2$, we have $\left \| f \right \|_{\mathcal{H}_1+\mathcal{H}_2} \leq \sqrt{\left \| f_1 \right \|_{\mathcal{H}_1}^2+\left \| f_2 \right \|_{\mathcal{H}_2}^2}$ with equality if, and only if $(f_1,f_2) \in O$. We can then deduce (2) $$\left \| f \right \|_{\mathcal{H}_1+\mathcal{H}_2} = \min_{(f_1,f_2)\in E, f_1+f_2=f} \sqrt{\left \| f_1 \right \|_{\mathcal{H}_1}^2+\left \| f_2 \right \|_{\mathcal{H}_2}^2}.$$

Reproducing property. In fact, to prove that $\mathcal{H}_1+\mathcal{H}_2$ endowed with the norm we just defined is the RKHS of $k_1+k_2$, we still need to prove the reproducing property: let $x \in \mathcal{X}$ and $f \in \mathcal{H}_1+\mathcal{H}_2$. We can write $f=\tilde{s}(f_1,f_2)$ and $k(x,\cdot)=\tilde{s}(A(x,\cdot),B(x,\cdot))$ where $(f_1,f_2)$ and $(A_x,B_x)$ live in $O$. Thus, $$\langle f,K_x \rangle_{\mathcal{H}_1+\mathcal{H}_2} = \langle(f_1,f_2) , (A(x,\cdot),B(x,\cdot)) \rangle_{E}=\langle(f_1,f_2) , \left(k_1(x,\cdot),k_2(x,\cdot)\right)+(A(x,\cdot)-k_1(x,\cdot),B(x,\cdot)-k_2(x,\cdot))\rangle_E$$ but, since $ A(x,\cdot)-k_1(x,\cdot)+B(x,\cdot)-k_2(x,\cdot)=0,$ the vector $(A_x-k_1(x,\cdot),B_x-k_2(x,\cdot))$ is orthogonal to every element in $O$, and in particular to $(f_1,f_2)$. Consequently, $$\langle f,K_x \rangle_{\mathcal{H}_1+\mathcal{H}_2} = \langle(f_1,f_2) , \left(k_1(x,\cdot),k_2(x,\cdot)\right) \rangle_E = f_1(x)+f_2(x)=f(x)$$ and the reproducing property is true.

$\mathcal{H}_1+\mathcal{H}_2$ is therefore the RKHS of $k_1+k_2$.

  • 1
    $\begingroup$ Nice proof. Nitpicking: wouldn't $N$ be $s^{-1}(\{0\})$ (extra braces)? $\endgroup$ – P. Camilleri Mar 9 '15 at 14:44
  • $\begingroup$ @M.Massias : yes! thanks! $\endgroup$ – PAM Mar 9 '15 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.