# How is $w=\lambda^{-1}X'(y-Xw)$ derived? [Ridge Regression]

In Ridge Regression we try to find the minimum of the following loss function:

$$\text{min}_w\mathcal{L}_{\lambda}(w,S)=\text{min}\lambda\|w\|^2+\sum^l_{i=1}(y_i-g(x_i))^2$$

Where:

• $$\lambda$$ is a positive number that defines the relative trade-off betweeen norm and loss
• $$\mathcal{L}$$ is the loss function
• $$w\in\mathbb{R}^n$$ is the vector of weights
• $$g(x_i)$$ is the predicted value of observation $$x_i$$

Taking the derivative of the cost function with respect to the parameters we obtain the equations (*)

$$X'Xw+\lambda w=(X'X+\lambda I_n)w=X'w$$

Where:

• $$I_n$$ is the $$n\times n$$ identity matrix
• $$X\in \mathbb{R}^{l\times n}$$ is the data matrix
• $$X'$$ is the transpose of $$X$$

The solution to the above equation is

$$w=(X'X+\lambda I_n)^{-1}X'y$$

Now, my book says that we can rewrite equations (*) in terms of $$w$$:

$$w=\lambda^{-1}X'(y-Xw)=X'\alpha$$

showing that $$w$$ can be written as a linear combination of the training points $$w=\sum^l_{i=1}\alpha_ix_i$$ with $$\alpha=\lambda^{-1}(y-Xw)$$

I have a hard time understanding how is $$w=\lambda^{-1}X'(y-Xw)$$ derived. Can someone show this algebraically?

• Assuming your $y$ is really a $w$, we have from equation (*) that $X'Xw + \lambda w = X' w$, so $\lambda w = X' w- X'Xw = X'(w-Xw)$ so $w = \lambda^{-1} X'(w-Xw)$
– jl00
Commented Aug 17, 2021 at 7:37
• @jl00 in the textbook it's written like that. Are you sure that's a typo? Commented Aug 17, 2021 at 7:42
• I don't see what else it could be... I'm not familiar with ridge regression, but from a purely algebraic standpoint I would bet that it's a typo.
– jl00
Commented Aug 17, 2021 at 7:47

Unfortunately equation (*) has a typo. You can tell there's a problem on the right hand side: the dimensions are wrong for $$X^\prime\in\mathbb{R}^{n\times l}$$ to multiply $$w\in\mathbb{R}^n$$.
We start from the objective function: $$\mathcal{L}(w) = ||y-Xw||^2 + \lambda||w||^2$$ where $$y\in\mathbb{R}^l$$, $$w\in\mathbb{R}^n$$ and $$X\in\mathbb{R}^{l\times n}$$. The derivative with respect to $$w$$ is given by $$\nabla_w\mathcal{L} = -2X^\prime(y-Xw) + 2\lambda w,$$ where $$X^\prime$$ is the transpose of $$X$$. Setting the gradient to zero immediately gives us the expression for $$w$$ which you were interested in: $$w = \frac{1}{\lambda}X^\prime(y-Xw).$$
To find the correct version of (*), we just collect the terms with $$w$$: $$(X^\prime X + \lambda I_n)w = X^\prime y,$$ which, when multiplied by the inverse of the left-hand matrix, leads us to the solution that you provided: $$w = (X^\prime X + \lambda I_n)^{-1}X^\prime y.$$