1
$\begingroup$

I've a question concerning the penalty term in an elastic net regression. In The elements of Statistical Learning by Hastie, Tibshirani & Friedman the formula (3.54) on p.73 says the penalty term is given by: $$ \lambda \cdot \displaystyle\sum_{j=1}^p (\alpha \cdot \beta_j^2 + (1 - \alpha) \cdot |\beta_j|) $$ This means for α=1 the formula transforms into a ridge regression. This is also consistent with the description in Zou & Hasti (2005, p. 303):

When α=1, the naive elastic net becomes simple ridge regression.

But almost every else (e.g., in the manual of the R package glmnet), it is written that ridge regression results if α=0. Actually, in the same book ("The elements...") on p. 681 in formula (18.20) it says, the penalty term has the form: $$ \displaystyle\sum_{j=1}^p (\alpha \cdot |\beta_j| + (1 - \alpha) \cdot \beta_j^2) $$

I wonder how this inconsistency can be explained. I would be grateful for any information.

Kind regards

Ulrich

$\endgroup$

1 Answer 1

0
$\begingroup$

I would not stress out over this. There isn't an established convention for representing how the elastic net penalty is parameterized (as evidenced by the different parameterizations within the same Elements of Statistical Learning text that you found), so you just need to be clear about how $\alpha$ is defined when discussing elastic net.

As a similar example, sometimes the LASSO objective function is expressed as $\sum_{i=1}^N (y_i - x_i^\top \beta)^2 + \lambda \|\beta\|_1$ while other times it is $\frac{1}{2}\sum_{i=1}^N (y_i - x_i^\top \beta)^2 + \lambda \|\beta\|_1$ or $\frac{1}{N}\sum_{i=1}^N (y_i - x_i^\top \beta)^2 + \lambda \|\beta\|_1$. Clearly the $\lambda$ in these different formulations are different, but it doesn't detract from discussion as long as everyone agrees on a definition beforehand.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.