# Ridge Regression and The Lasso

I'm trying to compare the penalties of ridge regression and the Lasso. I know the difference between the two methods and that ridge regression shrinks the regression coefficients towards zero while the Lasso sets some of them to be zero and that's why it's in a way a subset selection procedure. My question is that what should I compare in the penalties and is it a mathematical comparison?

My second question is: I obtained the form of the ridge regression estimates when the regressors are orthogonal. How should I obtain that of the Lasso when the regressors are orthogonal? Because there's no close form for the Lasso estimates.

Regarding the second question - this is one of the two (three?) cases when LASSO do have a closed form. Take $X$ an $n \times p$ matrix, hence you target function is $$S(\beta) = || Y - X \beta ||_2 ^ 2 + \lambda || \beta ||_1,$$ hence the gradient is $$\nabla S = 2 X ' ( Y - X \hat{\beta}) + \lambda \text{sign} (\hat{\beta}) = 0$$ where $\text{sign}(\hat{\beta})$ is an abuse of notation for vector of $\beta$'s OLS signs. So, as $x_i ^ Tx_j = 0$ and $x_i ^ T x_i = 1$, you have $$\hat{\beta}(\lambda) = X'Y+ \lambda/2 \text{sign}(\hat{\beta}) = \hat{\beta}^{OLS}+\lambda/2 \text{sign}(\hat{\beta}^{OLS}),$$ thus the ridge estimator for any $\beta_j$ can be written as $$\hat{\beta}_j(\lambda) = \text{sign}(\hat{\beta}^{OLS})\max\{\hat{\beta}_j^{OLS}+\lambda/2 \text{sign}(\hat{\beta}_j^{OLS}), 0\}.$$