2
$\begingroup$

In Ridge Regression we try to find the minimum of the following loss function:

$$\text{min}_w\mathcal{L}_{\lambda}(w,S)=\text{min}\lambda\|w\|^2+\sum^l_{i=1}(y_i-g(x_i))^2$$

Where:

  • $\lambda$ is a positive number that defines the relative trade-off betweeen norm and loss
  • $\mathcal{L}$ is the loss function
  • $w\in\mathbb{R}^n$ is the vector of weights
  • $g(x_i)$ is the predicted value of observation $x_i$

Taking the derivative of the cost function with respect to the parameters we obtain the equations (*)

$$X'Xw+\lambda w=(X'X+\lambda I_n)w=X'w$$

Where:

  • $I_n$ is the $n\times n$ identity matrix
  • $X\in \mathbb{R}^{l\times n}$ is the data matrix
  • $X'$ is the transpose of $X$

The solution to the above equation is

$$w=(X'X+\lambda I_n)^{-1}X'y$$

Now, my book says that we can rewrite equations (*) in terms of $w$:

$$w=\lambda^{-1}X'(y-Xw)=X'\alpha$$

showing that $w$ can be written as a linear combination of the training points $w=\sum^l_{i=1}\alpha_ix_i$ with $\alpha=\lambda^{-1}(y-Xw)$

I have a hard time understanding how is $w=\lambda^{-1}X'(y-Xw)$ derived. Can someone show this algebraically?

$\endgroup$
3
  • $\begingroup$ Assuming your $y$ is really a $w$, we have from equation (*) that $X'Xw + \lambda w = X' w$, so $\lambda w = X' w- X'Xw = X'(w-Xw)$ so $w = \lambda^{-1} X'(w-Xw)$ $\endgroup$
    – jl00
    Commented Aug 17, 2021 at 7:37
  • $\begingroup$ @jl00 in the textbook it's written like that. Are you sure that's a typo? $\endgroup$
    – Slim Shady
    Commented Aug 17, 2021 at 7:42
  • 1
    $\begingroup$ I don't see what else it could be... I'm not familiar with ridge regression, but from a purely algebraic standpoint I would bet that it's a typo. $\endgroup$
    – jl00
    Commented Aug 17, 2021 at 7:47

1 Answer 1

2
$\begingroup$

Unfortunately equation (*) has a typo. You can tell there's a problem on the right hand side: the dimensions are wrong for $X^\prime\in\mathbb{R}^{n\times l}$ to multiply $w\in\mathbb{R}^n$.

We start from the objective function: $$\mathcal{L}(w) = ||y-Xw||^2 + \lambda||w||^2$$ where $y\in\mathbb{R}^l$, $w\in\mathbb{R}^n$ and $X\in\mathbb{R}^{l\times n}$. The derivative with respect to $w$ is given by $$\nabla_w\mathcal{L} = -2X^\prime(y-Xw) + 2\lambda w,$$ where $X^\prime$ is the transpose of $X$. Setting the gradient to zero immediately gives us the expression for $w$ which you were interested in: $$ w = \frac{1}{\lambda}X^\prime(y-Xw). $$

To find the correct version of (*), we just collect the terms with $w$: $$ (X^\prime X + \lambda I_n)w = X^\prime y,$$ which, when multiplied by the inverse of the left-hand matrix, leads us to the solution that you provided: $$ w = (X^\prime X + \lambda I_n)^{-1}X^\prime y.$$

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .