The closest that "pure optimization" gets to regularization is probably the field of Robust Optimization, which aims to improve performance on unseen data by arming an adversary with the ability to perturb your solution ex-post. That is, you make a decision, then nature gets to selects the worst-case realization from some uncertainty set to change the problem slightly, but you are aware of what nature is going to do when you make your decision, so you make a better decision with respect to unseen data.
Indeed, in linear regression, the following robust linear regression problem:
\begin{align*}
\min_{w} \ \max_{\Delta: \Vert \Delta \Vert_F \leq \lambda} \ \frac{1}{\vert X \vert} \sum_x \Vert (\Delta+X)w-y\Vert_2^2,
\end{align*}
is directly equivalent to your original problem, by duality!
See this paper: https://www.sciencedirect.com/science/article/abs/pii/S0377221717302734 for more information on this.