Consider the class of functions $\mathcal H_\phi $ defined by $$\mathcal H_\phi :=\left\{h:\mathbb R^d \to \mathbb R\mid h(x) = \langle w,\phi(x)\rangle+b,\ (w,b)\in\mathbb R^D\times\mathbb R\right\} $$ Where $\phi : \mathbb R^d\to\mathbb R^D$ is a mapping function. Assume that we are given an empirical loss function $\hat L$ and we want to solve the following problem $$h : \arg\min_{\mathcal H_\phi} \hat L(h) \tag 1$$ Solving $(1)$ can be thought of as solving a soft-margin SVM problem.
To solve $(1)$ while avoiding overfitting, a common approach is to add a penalty term on the $\ell_2$ norm of the weight vector $w$, which is called $L_2$ or Tikhonov regularization : $$h_\lambda : \arg\min_{\mathcal H_\phi} \hat L(h) + \frac\lambda 2\|w\|^2\tag 2 $$ Alternatively, another approach is to directly constrain the weight vector to remain bounded in a certain radius. This is called Ivanov regularization : $$h_R : \arg\min_{\mathcal H_\phi} \hat L(h),\ \text{ s.t. } \|w\|^2\le R^2\tag 3$$ In their paper Tikhonov, Ivanov and Morozov regularization for support vector machine learning (2016), Oneto et al. proved, under the assumption that $\hat L$ is convex with respect to $w$, that a Tikhonov regularized SVM is equivalent to an Ivanov-regularized SVM, in the following sense :
Theorem : For any $\lambda$, there exists an $R \equiv R(\lambda)\in\mathbb R$ such that problems $(2)$ and $(3)$ have the same solutions (i.e. such that $h_\lambda = h_R$).
Regarding this theorem, I mainly have two questions :
- How does $R(\lambda)$ vary with respect to $\lambda$ ? Is it at least continuous ? Is it possible to get any lower bound on it in terms of $\lambda$ ? Intuitively, one would expect that $R(\lambda)\to\infty$ continuously as $\lambda \to 0$, but my attempts to prove it failed, and the proof given in the paper is not constructive so it doesn't help.
- Does this result apply to more general families of functions ? In particular, I am interested in Deep Neural Networks. In that case the convexity assumption on $\hat L$ does not hold anymore and it seems unlikely that the authors' approach will work. But intuitively again, one would expect that adding a penalty on the Euclidean norm of the weight parameters would correspond to restricting the parameters to remain in a ball whose radius depends on the strength of the penalty.
I will be grateful for any help or useful references you may provide.