Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

This is a part of a problem that I'm having, and I'm unclear how to do this particular step. I'm dealing with a ridged regression and I need to minimize the equation

$$\sum (Y_i - \beta^Tx_i)^2 + \lambda\sum\beta_j^2$$

The question says I need to find $\beta$ that minimizes this and I'm not sure how to do that because I have a vector.

How do I do this?

share|improve this question
    
What would you do, if $\beta$ wasn't a vector? –  sonystarmap Feb 25 '13 at 6:42
    
er just differentiate normally? –  praks5432 Feb 25 '13 at 7:14
1  
Exactly, differentiate and set it equal to zero. Now that you have a vector, just do that for each component and you should get a system of equations to solve. –  sonystarmap Feb 25 '13 at 7:17
    
wait but how then do I deal with the beta i in the second sigma? –  praks5432 Feb 25 '13 at 7:19
    
Hint $\sum_j\beta_j^2=\mathbf{\beta}^T\mathbf{\beta}$. –  Daryl Feb 25 '13 at 7:53

1 Answer 1

Lets us define $$ E = \sum (Y_i -\beta^T x_i)^2+ \lambda \sum \beta_j^2$$ And we want to find optimal values for $\beta$. Thus we take the derivative of $E$ with respect to $\beta$ (Note that $x_{ij}$ means component $j$ of vector $x_i$). \begin{align} \nabla_\beta E &= \begin{pmatrix} \frac{\partial}{\partial \beta_1}\\ ..\\ \frac{\partial}{\partial \beta_n}\end{pmatrix}E \\ &= \begin{pmatrix} \frac{\partial}{\partial \beta_1}(\sum (Y_i -\beta^T x_i)^2+ \lambda \sum \beta_j^2)\\ ..\\ \frac{\partial}{\partial \beta_n}(\sum (Y_i -\beta^T x_i)^2+ \lambda \sum \beta_j^2) \end{pmatrix} \\ &= \begin{pmatrix} \sum -2x_{i1}(Y_i -\beta^T x_i)+ 2\lambda \beta_1\\ ..\\ \sum -2x_{in}(Y_i -\beta^T x_i)+ 2\lambda \beta_n \end{pmatrix} \\ &= \begin{pmatrix} \sum -2x_{i1}Y_i +\sum 2x_{i1} \beta^T x_i+ 2\lambda \beta_1\\ ..\\ \sum -2x_{in}Y_i +\sum 2x_{in} \beta^T x_i+ 2\lambda \beta_n\end{pmatrix} \\ &= \begin{pmatrix} \sum -2x_{i1}Y_i +2\sum \beta_j \sum x_{i1} x_{ij}+ 2\lambda \beta_1\\ ..\\ \sum -2x_{in}Y_i +2\sum \beta_j \sum x_{in} x_{ij}+ 2\lambda \beta_n\end{pmatrix} \\ &= A\beta +r \end{align} Where \begin{align} A = 2\begin{pmatrix} \lambda +\sum x_{i1}x_{i1} & \sum x_{i1}x_{i2} & \sum x_{i1}x_{i3} & ... & \sum x_{i1}x_{in} \\ \sum x_{i2}x_{i1} & \lambda + \sum x_{i2}x_{i2} & \sum x_{i2}x_{i3} & ... & \sum x_{i2}x_{in} \\ ... & ... & ... & ... & ... & \\ \sum x_{in}x_{i1} & \sum x_{in}x_{i2} & \sum x_{in}x_{i3} & ... &\lambda+ \sum x_{in}x_{in} \end{pmatrix} \end{align} And \begin{align} r = -2\begin{pmatrix}\sum x_{i1}Y_i \\ ..\\\sum x_{in}Y_i \end{pmatrix} \end{align} To obtain a minima, we need to solve $A\beta+r=0$. This gives you $\beta$.

I really encourage you, to check my indices and sums, as I did this rather quick.

share|improve this answer
    
I would appreciate it, if someone could doble check my calculation. Thanks in advance! –  sonystarmap Feb 25 '13 at 8:48

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.