# Orthogonalisation in Reproducing Kernel Hilbert Space (RKHS) and null space

Let $$\mathcal{X}$$ be a set equipped with a positive definite kernel $$K$$ with value $$K(x,\, x')$$. Let $$\mathcal{K}$$ be the corresponding RKHS and consider a closed linear subspace $$\mathcal{F}$$ in $$\mathcal{K}$$ and $$\mathcal{G}$$ be the orthogonal supplement of $$\mathcal{F}$$ in $$\mathcal{K}$$. Beside the the inner product $$\langle\,, \,\rangle_{\mathcal{K}}$$ we can define a new inner product $$\langle\,, \,\rangle_{\widetilde{\mathcal{K}}}$$ on $$\mathcal{K}$$ by

$$\left\langle h, \, h' \right\rangle_{\widetilde{\mathcal{K}}} := \left\langle P.h, \, P.h' \right\rangle_{\mathcal{K}}$$

where $$P.h$$ denotes the orthogonal projection on $$\mathcal{G}$$. Then $$\mathcal{H}$$ equipped with $$\langle\,, \,\rangle_{\widetilde{\mathcal{K}}}$$ becomes a new RKHS with null space $$\mathcal{F}$$. How can we relate the semi-definite positive kernel $$\widetilde{K}$$ on $$\mathcal{X}$$ to the original kernel $$K$$?

Comments. In the RKHS framework for splines the original kernel is often taken as $$K(x, \, x') = F(x,\,x') + G(x,\,x')$$ where $$F$$ and $$G$$ are semi-positive definite kernels with RKHS $$\mathcal{F}$$ and $$\mathcal{G}$$ which form a direct sum of $$\mathcal{K}$$ hence we simply have $$\widetilde{K}(x, \, x') = G(x, \, x')$$ in this case. The null space $$\mathcal{F}$$ is then usually finite-dimensional. Rather than starting from a kernel which is a direct sum, I would like to start from a general kernel hence to achieve some kind of subtraction of kernels. Of special interest is the case where $$\mathcal{F}$$ is finite-dimensional and an example with a closed form for $$K(x, \, x')$$ and $$\widetilde{K}(x, \, x')$$ would be great. The question relates to universal Kriging applications where $$\mathcal{X}$$ is a domain in $$\mathbb{R}^d$$, $$\mathcal{F}$$ is spanned by so-called trend functions and $$K$$ is a kernel given in closed form: Whittle-Matérn, square-exponential, ...

• I have trouble to understand your question "can we relate the kernels". You state the relation between $K$ and $\tilde K$ correctly and subtraction is also straigthforward under your assumptions (the two subspaces being closed and orthogonal). So, what else do you want?
– g g
Dec 17, 2021 at 10:53
• Using for instance the exponential kernel $K(x, \,x') = \exp\{-\kappa |x -x'|\}$ on a real interval $[0, \, T]$, the RKHS and its norm are known. If I take $\mathcal{F}$ as generated by the constant function $1$, what is the expression of $\widetilde{K}(x,\,x')$ and what are the steps in its derivation? I found $\widetilde{K} = K - 2 / (2 + \kappa T)$ which seems strange and I think that my derivation was wrong. Also I believe that $\widetilde{K}$ can not be positive definite.
– Yves
Dec 17, 2021 at 14:38

The identity map of the Hilbert Space $$\mathscr{H}$$ can be written as $$I=P + (I-P)$$ where $$P$$ and $$I-P$$ are the orthogonal projections on your spaces $$\mathscr{F}$$ and $$\mathscr{G}$$. Now set $$k_x(\cdot)=K(x,\cdot)$$ then due to orthogonality
$$K(x,y) = \langle k_x,k_y\rangle = \langle Pk_x,Pk_y\rangle + \langle (I-P)k_x, (I-P)k_y\rangle.$$ and nothing holds you back from simply writing $$\tilde K(x,y) = K(x,y) - \langle(I-P)k_x,(I-P)k_y\rangle.$$ Note that the RHS is always non-negative since projections reduce norms, i.e. $$\lVert (I-P)k_x\rVert\leq \lVert k_x\rVert.$$

Now to your concrete example. The orthogonal projection of the function $$k_x$$ on the complement of the space spanned by the constant function $$1$$ is: $$\tilde{k}_x = k_x - \frac{\langle k_x,1\rangle}{\lVert 1 \rVert^2}1$$ and $$\tilde K(x,y) = \langle \tilde{k}_x, \tilde{k}_y\rangle = \langle k_x - \frac{\langle k_x,1\rangle}{\lVert 1 \rVert^2}1, k_y - \frac{\langle k_y,1\rangle}{\lVert 1 \rVert^2}1\rangle = \langle {k}_x, {k}_y\rangle - \frac{\langle {k}_x, 1\rangle \langle {k}_y, 1\rangle}{\lVert 1\rVert^2}.$$

Assume $$\tau=1$$ for simplicity and note that the inner product of your space is $$\langle f,g \rangle = f(0)g(0) + f(T)g(T) + \int_0^T f(t)g(t)\,dt + \int_0^T f'(t)g'(t)\,dt$$ which means the two inner products can be evaluated using $$\langle k_x,1 \rangle = k_x(0) + k_x(T) + \int_0^T k_x(t)\,dt.$$

By using Gram-Schmidt the above works for projections on any finite dimensional space.

• I am quite uncomfortable with the notations in the first two equations since $Px$ is only clear for me when $x$ is a function in the RKHS. I can not figure out what the norm $\|x \|$ is for $x \in \mathcal{X}$ which is not even a linear space. I think the right formalism is provided by Theorem 11 in the book by Berlinet and Thomas-Agnan. Also I realized that we must restrict to $\mathcal{G}$ to define a RKHS otherwise we get as semi-Hilbert space, not a Hibert space.
– Yves
Dec 20, 2021 at 12:43
• Of course, you are right and I corrected the notation. With respect to the restrictions: Since you assumed that $\mathscr{F}$ is closed, which it is in particular if it is finite dimensional, you do not have to put any additional requirements on $\mathscr{G}.$ This is no surprise in the finite dimensional case, since you can write down its Kernel explicitly.
– g g
Dec 20, 2021 at 13:07