0
$\begingroup$

I'm trying to compare the penalties of ridge regression and the Lasso. I know the difference between the two methods and that ridge regression shrinks the regression coefficients towards zero while the Lasso sets some of them to be zero and that's why it's in a way a subset selection procedure. My question is that what should I compare in the penalties and is it a mathematical comparison?

My second question is: I obtained the form of the ridge regression estimates when the regressors are orthogonal. How should I obtain that of the Lasso when the regressors are orthogonal? Because there's no close form for the Lasso estimates.

$\endgroup$
1
$\begingroup$

Ridge regression uses a quadratic penalty, and this is very small for small coefficients, giving almost no regularisation benefit from further reduction, which is why it typically fails to force any of them fully to zero

LASSO uses an absolute value penalty, so reducing a small coefficient to zero can give the same regularisation benefit as reducing a large coefficient by a small amount, making the approach frequently reduce the number of variables actively used in the final model

For your second question, it might be better to demonstrate LASSO with actual data

$\endgroup$
0
$\begingroup$

Regarding the second question - this is one of the two (three?) cases when LASSO do have a closed form. Take $ X $ an $n \times p$ matrix, hence you target function is $$ S(\beta) = || Y - X \beta ||_2 ^ 2 + \lambda || \beta ||_1, $$ hence the gradient is $$ \nabla S = 2 X ' ( Y - X \hat{\beta}) + \lambda \text{sign} (\hat{\beta}) = 0 $$ where $\text{sign}(\hat{\beta})$ is an abuse of notation for vector of $\beta$'s OLS signs. So, as $x_i ^ Tx_j = 0$ and $x_i ^ T x_i = 1$, you have $$ \hat{\beta}(\lambda) = X'Y+ \lambda/2 \text{sign}(\hat{\beta}) = \hat{\beta}^{OLS}+\lambda/2 \text{sign}(\hat{\beta}^{OLS}), $$ thus the ridge estimator for any $\beta_j$ can be written as $$ \hat{\beta}_j(\lambda) = \text{sign}(\hat{\beta}^{OLS})\max\{\hat{\beta}_j^{OLS}+\lambda/2 \text{sign}(\hat{\beta}_j^{OLS}), 0\}. $$

This is a special case as for orthogonal design each estimator does not depends on the other variables/coefficients.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.