1
$\begingroup$

In one of the texts that I am reading it is given that regularization parameter restricts the choice of functions in case data given is not sufficient for processing of signal. It is given that lambda acts as relative trade-off between norm and loss function. I have two questions:

  1. Why we are choosing norm as the criterion in this optimization problem?
  2. How lambda is acting as a trade-off?

My understanding says that as I increase lambda my weight has to be reduced much more as compared to when lambda is low,which happens only when we restrict lower limit of the optimization problem.So how lambda is a trade-off when both norm and loss-function reduces with increase of lambda.

Please shed some light on this.

Thank you.

Ridge Regression

$\endgroup$

1 Answer 1

0
$\begingroup$

Why we are choosing norm as the criterion in this optimization problem?

Mainly because it has a closed form solution (unlike $l_1$ norm penalty) since it was proposed much before the intensive computing era. There is nothing special in Euclidean norm. The $l_1$ norm is much more popular these days, unlike the former, it also performs variables ("features") selection and not only shrinks the magnitude of the coefficients.

How lambda is acting as a trade-off?

For large penalty ($\lambda$) you shrink heavily the coefficients of the features, namely you don't put much faith on your raw data. For $\lambda \to \infty$, the coefficients are shrank to zero. For $\lambda = 0$, you just use the usual least squares estimates, i.e., you use only the loss function and not penalizing it or stabilizing the results.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .