1
$\begingroup$

I am reading The Elements of Statistical Learning (Hastie et. al.) and, having already derived the Reinsch form $S_\lambda = (\mathbb{I} + \lambda K)^{-1}$ for a smoothing spline, we are asked to:

Prove that for a smoothing spline the null space of $K$ is spanned by functions linear in $X$

(where $K = N^{T}\Omega_NN$).

What is the proof for this and more importantly what is its significance?

Reference: Chapter 5: Basis Expansions and Regularisation

$\endgroup$

1 Answer 1

1
$\begingroup$

Below is a proof of the claim.

Notation:

  • $x_i$, $i=1,\ldots, N$ is a data set
  • $N_i(X)$, $i=1,\ldots, N$ is a basis for the space of natural cubic splines with knots at the $x_1,\ldots ,x_N$
  • $\mathbf{N}$ is the invertible $N\times N$ matrix with $\mathbf{N}_{i,j}=N_j(x_i)$
  • $\mathbf{\Omega}_N$ is the $N\times N$ matrix with $(\mathbf{\Omega}_N)_{j,k} = \int_{\mathbb{R}} N_j^{\prime\prime}(t)N_k^{\prime\prime}(t) \,\mathrm{d}t$

I think that you should have found $\mathbf{K} = (\mathbf{N}^{-1})^T\mathbf{\Omega}_N\mathbf{N}^{-1}$ rather than $\mathbf{N}^T\mathbf{\Omega}_N\mathbf{N}$.

Suppose that $\mathbf{u}=(u_1,\ldots ,u_N)$ lies in the null space of $\mathbf{K}$. Since the space of natural cubic splines is $N$-dimensional, we can find one that interpolates the pairs $\{(x_i, u_i)\}_{i=1}^N$. That is, there exists $\beta=(\beta_1,\ldots,\beta_N)$ such that $u_i=\sum_{j=1}^N \beta_j N_j(x_i) \Rightarrow \mathbf{u}=\mathbf{N}\beta$. We can also think of $\mathbf{u}$ as the evaluation of the function $u(X)=\sum_{j=1}^N \beta_k N_j(X)$ at the data points $x_1,\ldots ,x_N$.

Since $\mathbf{u}$ is in the null space of $\mathbf{K}=(\mathbf{N}^{-1})^T\Omega_N\mathbf{N}^{-1}$,

\begin{equation} 0 = \mathbf{K}\mathbf{u} = \mathbf{K}\mathbf{N}\beta \qquad\Rightarrow\qquad \Omega_N\beta = 0. \end{equation}

From the definition, the $j$th entry of $\Omega_N\beta$ is

\begin{equation} \sum_{k=1}^N (\Omega_{jk})_N \beta_k = \int_{\mathbb{R}} \sum_{k=1}^N \beta_k N_j^{\prime\prime}(t)N_k^{\prime\prime}(t) \,\mathrm{d}t = \int_{\mathbb{R}} N_j^{\prime\prime}(t)u^{\prime\prime}(t) \,\mathrm{d}t. \end{equation}

Since this is zero for all $j$, $\int_{\mathbb{R}} s^{\prime\prime}(t)u^{\prime\prime}(t) \,\mathrm{d}t=0$ for any natural cubic spline $s(t)$. In particular this is true for $s(t)=u(t)$, so $u^{\prime\prime}(t)$ is identically zero. Since $u(t)$ is a cubic spline, this implies that it is linear.


As for the significance, it shows that regularisation via the smoother matrix $\mathbf{S}_{\lambda}$ doesn't shrink linear functions of $X$. This is reassuring as we want to penalise components that are too 'rough', not the linear part. This also implies that the linear component of the smoothed spline $\hat{f}(x)$ is equal to the least squares line fit, since it is independent of $\lambda$ and $\hat{f}(x)$ equals the least squares fit for $\lambda=\infty$.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .