Regularization: Optimization with Non-Linear Constraints?

Question

I have the following question on "Regularization vs. Constrained Optimization" :

In the context of statistical modelling and machine learning, we are often taught about "Regularization" as a method of dealing with the "Bias-Variance Tradeoff" (i.e. stabilizing the inconsistent performance of complicated models). When a L1-Norm or L2-Norm Penalty Term is added to the estimation function (corresponding to the statistical model) being optimized, some of the model parameters will either "shrink" in size towards 0, thus producing a "sparser" model that is more likely to retain its "low bias" but possible reduce its "high variance":

I have often heard of functions containing these L1-Norm and L2-Norm "Penalty Terms" being referred to as "optimization constraints" (i.e. the "feasible region" from which valid choices of model parameters can belong to has now been "altered" due to these "norm penalty constraints"):

My Question: When we estimate some statistical model's parameters and the estimation equation contains some "regularization penalty term," would it be incorrect to refer to this as an example of "constrained optimization"?

I understand that in the traditional sense, some people consider that "Regularization" can not be considered as "Constrained Optimization", because the "Regularization Parameter" (lambda) is not "fixed" - and several values of "lambda" are considered (i.e. "cross validation") and the optimization process is repeated for all these values of "lambda" (a final value of "lambda" is selected based on some other criteria, e.g. which value of "lambda" results in the machine learning model having the lowest overall "error").

I also realize that in machine learning applications we are not necessarily interested in finding the "true optimum point" of the function we are optimizing, a "good enough point" is usually acceptable due to the computational complexity.

But suppose we were to select only a single value of "lambda" - is Regularization any different from a solving an Optimization Problem with a Non-Linear Constraint?

Is regularized optimization in Machine Learning and Statistical Modelling fundamentally any different (with the exception of usually being more difficult and solved using approximate stochastic iterative methods) from Constrained Optimization?

Thanks!

References:

• https://en.wikipedia.org/wiki/Regularization_(mathematics)
• http://ab-initio.mit.edu/wiki/index.php?title=NLopt_Tutorial

Answer 1

Fixing $\lambda$ wouldn't turn this into a constrained optimisation problem. It would become a constrained optimisation problem if you imposed a condition like "minimise $\sum \left(t(\mathbf{x}_j)-\sum w_i h_i(\mathbf{x}_j)\right)^2$, subject to $\sum |w_i|=c$". People do this sometimes, and this constrained problem has the same solution as the unconstrained problem if you choose $c$ corresponding to the minimiser for the unconstrained problem (since any solution to the constrained problem would give you a feasible point for the unconstrained problem with the same penalty term). But they are different problems. You would use a different approach to minimise the unconstrained problem with a fixed $\lambda$ than you would use to solve the constrained problem with a fixed $c$.