I want to comprehend the derivative of the cost function in linear regression involving Ridge regularization, the equation is:

$$L^{\text{Ridge}}(\beta) = \sum_{i=1}^n (y_i - \phi(x_i)^T\beta)^2 + \lambda \sum_{j=1}^k \beta_j^2$$

Where the sum of squares can be rewritten as:

$$L^{}(\beta) = ||y-X\beta||^2 + \lambda \sum_{j=1}^k \beta_j^2$$

For finding the optimum its derivative is set to zero, which leads to this solution:

$$\beta^{\text{Ridge}} = (X^TX + \lambda I)^{-1} X^T y$$

Now I would like to understand this and try to derive it myself, heres what I got:

Since $||x||^2 = x^Tx$ and $\frac{\partial}{\partial x} [x^Tx] = 2x^T$ this can be applied by using the chain rule:

\begin{align*} \frac{\partial}{\partial \beta} L^{\text{Ridge}}(\beta) = 0^T &= -2(y - X \beta)^TX + 2 \lambda I\\ 0 &= -2(y - X \beta) X^T + 2 \lambda I\\ 0 &= -2X^Ty + 2X^TX\beta + 2 \lambda I\\ 0 &= -X^Ty + X^TX\beta + 2 \lambda I\\ &= X^TX\beta + 2 \lambda I\\ (X^TX + \lambda I)^{-1} X^Ty &= \beta \end{align*}

Where I strugle is the next-to-last equation, I multiply it with $(X^TX + \lambda I)^{-1}$ and I don't think that leads to a correct equation.

What have I done wrong?


1 Answer 1


You have differentiated $L$ incorrectly, specifically the $\lambda ||\beta||^2$ term. The correct expression is: $\frac{\partial L(\beta)}{\partial \beta} = 2(( X \beta - y)^T X + \lambda \beta^T)$, from which the desired result follows by equating to zero and taking transposes.

  • $\begingroup$ Thanks, I had this intuition already, that I can rewrite the second sum to $\lambda||\beta||^2$, for what soever reason I dropped it. I think you forgot a minus in there: $$2(( X \beta - y)^T (-X) + \lambda \beta^T)$$ Since the inner derivative of $X \beta - y$ is $-X$, isn't it? $\endgroup$
    – Mahoni
    Jul 2, 2012 at 13:58
  • $\begingroup$ The signs are correct. $\endgroup$
    – copper.hat
    Jul 2, 2012 at 14:56
  • $\begingroup$ ah I see that now, thanks $\endgroup$
    – Mahoni
    Jul 2, 2012 at 15:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .