Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

This is a part of a problem that I'm having, and I'm unclear how to do this particular step. I'm dealing with a ridged regression and I need to minimize the equation

$$\sum (Y_i - \beta^Tx_i)^2 + \lambda\sum\beta_j^2$$

The question says I need to find $\beta$ that minimizes this and I'm not sure how to do that because I have a vector.

How do I do this?

share|cite|improve this question
What would you do, if $\beta$ wasn't a vector? – k1next Feb 25 '13 at 6:42
er just differentiate normally? – praks5432 Feb 25 '13 at 7:14
Exactly, differentiate and set it equal to zero. Now that you have a vector, just do that for each component and you should get a system of equations to solve. – k1next Feb 25 '13 at 7:17
wait but how then do I deal with the beta i in the second sigma? – praks5432 Feb 25 '13 at 7:19
Hint $\sum_j\beta_j^2=\mathbf{\beta}^T\mathbf{\beta}$. – Daryl Feb 25 '13 at 7:53

Lets us define $$ E = \sum (Y_i -\beta^T x_i)^2+ \lambda \sum \beta_j^2$$ And we want to find optimal values for $\beta$. Thus we take the derivative of $E$ with respect to $\beta$ (Note that $x_{ij}$ means component $j$ of vector $x_i$). \begin{align} \nabla_\beta E &= \begin{pmatrix} \frac{\partial}{\partial \beta_1}\\ ..\\ \frac{\partial}{\partial \beta_n}\end{pmatrix}E \\ &= \begin{pmatrix} \frac{\partial}{\partial \beta_1}(\sum (Y_i -\beta^T x_i)^2+ \lambda \sum \beta_j^2)\\ ..\\ \frac{\partial}{\partial \beta_n}(\sum (Y_i -\beta^T x_i)^2+ \lambda \sum \beta_j^2) \end{pmatrix} \\ &= \begin{pmatrix} \sum -2x_{i1}(Y_i -\beta^T x_i)+ 2\lambda \beta_1\\ ..\\ \sum -2x_{in}(Y_i -\beta^T x_i)+ 2\lambda \beta_n \end{pmatrix} \\ &= \begin{pmatrix} \sum -2x_{i1}Y_i +\sum 2x_{i1} \beta^T x_i+ 2\lambda \beta_1\\ ..\\ \sum -2x_{in}Y_i +\sum 2x_{in} \beta^T x_i+ 2\lambda \beta_n\end{pmatrix} \\ &= \begin{pmatrix} \sum -2x_{i1}Y_i +2\sum \beta_j \sum x_{i1} x_{ij}+ 2\lambda \beta_1\\ ..\\ \sum -2x_{in}Y_i +2\sum \beta_j \sum x_{in} x_{ij}+ 2\lambda \beta_n\end{pmatrix} \\ &= A\beta +r \end{align} Where \begin{align} A = 2\begin{pmatrix} \lambda +\sum x_{i1}x_{i1} & \sum x_{i1}x_{i2} & \sum x_{i1}x_{i3} & ... & \sum x_{i1}x_{in} \\ \sum x_{i2}x_{i1} & \lambda + \sum x_{i2}x_{i2} & \sum x_{i2}x_{i3} & ... & \sum x_{i2}x_{in} \\ ... & ... & ... & ... & ... & \\ \sum x_{in}x_{i1} & \sum x_{in}x_{i2} & \sum x_{in}x_{i3} & ... &\lambda+ \sum x_{in}x_{in} \end{pmatrix} \end{align} And \begin{align} r = -2\begin{pmatrix}\sum x_{i1}Y_i \\ ..\\\sum x_{in}Y_i \end{pmatrix} \end{align} To obtain a minima, we need to solve $A\beta+r=0$. This gives you $\beta$.

I really encourage you, to check my indices and sums, as I did this rather quick.

share|cite|improve this answer
I would appreciate it, if someone could doble check my calculation. Thanks in advance! – k1next Feb 25 '13 at 8:48

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.