I am reading The Elements of Statistical Learning (Hastie et. al.) and, having already derived the Reinsch form $S_\lambda = (\mathbb{I} + \lambda K)^{-1}$ for a smoothing spline, we are asked to:

Prove that for a smoothing spline the null space of $K$ is spanned by functions linear in $X$

(where $K = N^{T}\Omega_NN$).

What is the proof for this and more importantly what is its significance?

Reference: Chapter 5: Basis Expansions and Regularisation


Below is a proof of the claim.


  • $x_i$, $i=1,\ldots, N$ is a data set
  • $N_i(X)$, $i=1,\ldots, N$ is a basis for the space of natural cubic splines with knots at the $x_1,\ldots ,x_N$
  • $\mathbf{N}$ is the invertible $N\times N$ matrix with $\mathbf{N}_{i,j}=N_j(x_i)$
  • $\mathbf{\Omega}_N$ is the $N\times N$ matrix with $(\mathbf{\Omega}_N)_{j,k} = \int_{\mathbb{R}} N_j^{\prime\prime}(t)N_k^{\prime\prime}(t) \,\mathrm{d}t$

I think that you should have found $\mathbf{K} = (\mathbf{N}^{-1})^T\mathbf{\Omega}_N\mathbf{N}^{-1}$ rather than $\mathbf{N}^T\mathbf{\Omega}_N\mathbf{N}$.

Suppose that $\mathbf{u}=(u_1,\ldots ,u_N)$ lies in the null space of $\mathbf{K}$. Since the space of natural cubic splines is $N$-dimensional, we can find one that interpolates the pairs $\{(x_i, u_i)\}_{i=1}^N$. That is, there exists $\beta=(\beta_1,\ldots,\beta_N)$ such that $u_i=\sum_{j=1}^N \beta_j N_j(x_i) \Rightarrow \mathbf{u}=\mathbf{N}\beta$. We can also think of $\mathbf{u}$ as the evaluation of the function $u(X)=\sum_{j=1}^N \beta_k N_j(X)$ at the data points $x_1,\ldots ,x_N$.

Since $\mathbf{u}$ is in the null space of $\mathbf{K}=(\mathbf{N}^{-1})^T\Omega_N\mathbf{N}^{-1}$,

\begin{equation} 0 = \mathbf{K}\mathbf{u} = \mathbf{K}\mathbf{N}\beta \qquad\Rightarrow\qquad \Omega_N\beta = 0. \end{equation}

From the definition, the $j$th entry of $\Omega_N\beta$ is

\begin{equation} \sum_{k=1}^N (\Omega_{jk})_N \beta_k = \int_{\mathbb{R}} \sum_{k=1}^N \beta_k N_j^{\prime\prime}(t)N_k^{\prime\prime}(t) \,\mathrm{d}t = \int_{\mathbb{R}} N_j^{\prime\prime}(t)u^{\prime\prime}(t) \,\mathrm{d}t. \end{equation}

Since this is zero for all $j$, $\int_{\mathbb{R}} s^{\prime\prime}(t)u^{\prime\prime}(t) \,\mathrm{d}t=0$ for any natural cubic spline $s(t)$. In particular this is true for $s(t)=u(t)$, so $u^{\prime\prime}(t)$ is identically zero. Since $u(t)$ is a cubic spline, this implies that it is linear.

As for the significance, it shows that regularisation via the smoother matrix $\mathbf{S}_{\lambda}$ doesn't shrink linear functions of $X$. This is reassuring as we want to penalise components that are too 'rough', not the linear part. This also implies that the linear component of the smoothed spline $\hat{f}(x)$ is equal to the least squares line fit, since it is independent of $\lambda$ and $\hat{f}(x)$ equals the least squares fit for $\lambda=\infty$.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.