# Prove the null space of K is spanned by functions linear in X for a smoothing spline

I am reading The Elements of Statistical Learning (Hastie et. al.) and, having already derived the Reinsch form $$S_\lambda = (\mathbb{I} + \lambda K)^{-1}$$ for a smoothing spline, we are asked to:

Prove that for a smoothing spline the null space of $$K$$ is spanned by functions linear in $$X$$

(where $$K = N^{T}\Omega_NN$$).

What is the proof for this and more importantly what is its significance?

Reference: Chapter 5: Basis Expansions and Regularisation

Below is a proof of the claim.

Notation:

• $$x_i$$, $$i=1,\ldots, N$$ is a data set
• $$N_i(X)$$, $$i=1,\ldots, N$$ is a basis for the space of natural cubic splines with knots at the $$x_1,\ldots ,x_N$$
• $$\mathbf{N}$$ is the invertible $$N\times N$$ matrix with $$\mathbf{N}_{i,j}=N_j(x_i)$$
• $$\mathbf{\Omega}_N$$ is the $$N\times N$$ matrix with $$(\mathbf{\Omega}_N)_{j,k} = \int_{\mathbb{R}} N_j^{\prime\prime}(t)N_k^{\prime\prime}(t) \,\mathrm{d}t$$

I think that you should have found $$\mathbf{K} = (\mathbf{N}^{-1})^T\mathbf{\Omega}_N\mathbf{N}^{-1}$$ rather than $$\mathbf{N}^T\mathbf{\Omega}_N\mathbf{N}$$.

Suppose that $$\mathbf{u}=(u_1,\ldots ,u_N)$$ lies in the null space of $$\mathbf{K}$$. Since the space of natural cubic splines is $$N$$-dimensional, we can find one that interpolates the pairs $$\{(x_i, u_i)\}_{i=1}^N$$. That is, there exists $$\beta=(\beta_1,\ldots,\beta_N)$$ such that $$u_i=\sum_{j=1}^N \beta_j N_j(x_i) \Rightarrow \mathbf{u}=\mathbf{N}\beta$$. We can also think of $$\mathbf{u}$$ as the evaluation of the function $$u(X)=\sum_{j=1}^N \beta_k N_j(X)$$ at the data points $$x_1,\ldots ,x_N$$.

Since $$\mathbf{u}$$ is in the null space of $$\mathbf{K}=(\mathbf{N}^{-1})^T\Omega_N\mathbf{N}^{-1}$$,

$$$$0 = \mathbf{K}\mathbf{u} = \mathbf{K}\mathbf{N}\beta \qquad\Rightarrow\qquad \Omega_N\beta = 0.$$$$

From the definition, the $$j$$th entry of $$\Omega_N\beta$$ is

$$$$\sum_{k=1}^N (\Omega_{jk})_N \beta_k = \int_{\mathbb{R}} \sum_{k=1}^N \beta_k N_j^{\prime\prime}(t)N_k^{\prime\prime}(t) \,\mathrm{d}t = \int_{\mathbb{R}} N_j^{\prime\prime}(t)u^{\prime\prime}(t) \,\mathrm{d}t.$$$$

Since this is zero for all $$j$$, $$\int_{\mathbb{R}} s^{\prime\prime}(t)u^{\prime\prime}(t) \,\mathrm{d}t=0$$ for any natural cubic spline $$s(t)$$. In particular this is true for $$s(t)=u(t)$$, so $$u^{\prime\prime}(t)$$ is identically zero. Since $$u(t)$$ is a cubic spline, this implies that it is linear.

As for the significance, it shows that regularisation via the smoother matrix $$\mathbf{S}_{\lambda}$$ doesn't shrink linear functions of $$X$$. This is reassuring as we want to penalise components that are too 'rough', not the linear part. This also implies that the linear component of the smoothed spline $$\hat{f}(x)$$ is equal to the least squares line fit, since it is independent of $$\lambda$$ and $$\hat{f}(x)$$ equals the least squares fit for $$\lambda=\infty$$.