# Minimizing a function with vectors

This is a part of a problem that I'm having, and I'm unclear how to do this particular step. I'm dealing with a ridged regression and I need to minimize the equation

$$\sum (Y_i - \beta^Tx_i)^2 + \lambda\sum\beta_j^2$$

The question says I need to find $\beta$ that minimizes this and I'm not sure how to do that because I have a vector.

How do I do this?

-
What would you do, if $\beta$ wasn't a vector? – k1next Feb 25 '13 at 6:42
er just differentiate normally? – praks5432 Feb 25 '13 at 7:14
Exactly, differentiate and set it equal to zero. Now that you have a vector, just do that for each component and you should get a system of equations to solve. – k1next Feb 25 '13 at 7:17
wait but how then do I deal with the beta i in the second sigma? – praks5432 Feb 25 '13 at 7:19
Hint $\sum_j\beta_j^2=\mathbf{\beta}^T\mathbf{\beta}$. – Daryl Feb 25 '13 at 7:53

Lets us define $$E = \sum (Y_i -\beta^T x_i)^2+ \lambda \sum \beta_j^2$$ And we want to find optimal values for $\beta$. Thus we take the derivative of $E$ with respect to $\beta$ (Note that $x_{ij}$ means component $j$ of vector $x_i$). \begin{align} \nabla_\beta E &= \begin{pmatrix} \frac{\partial}{\partial \beta_1}\\ ..\\ \frac{\partial}{\partial \beta_n}\end{pmatrix}E \\ &= \begin{pmatrix} \frac{\partial}{\partial \beta_1}(\sum (Y_i -\beta^T x_i)^2+ \lambda \sum \beta_j^2)\\ ..\\ \frac{\partial}{\partial \beta_n}(\sum (Y_i -\beta^T x_i)^2+ \lambda \sum \beta_j^2) \end{pmatrix} \\ &= \begin{pmatrix} \sum -2x_{i1}(Y_i -\beta^T x_i)+ 2\lambda \beta_1\\ ..\\ \sum -2x_{in}(Y_i -\beta^T x_i)+ 2\lambda \beta_n \end{pmatrix} \\ &= \begin{pmatrix} \sum -2x_{i1}Y_i +\sum 2x_{i1} \beta^T x_i+ 2\lambda \beta_1\\ ..\\ \sum -2x_{in}Y_i +\sum 2x_{in} \beta^T x_i+ 2\lambda \beta_n\end{pmatrix} \\ &= \begin{pmatrix} \sum -2x_{i1}Y_i +2\sum \beta_j \sum x_{i1} x_{ij}+ 2\lambda \beta_1\\ ..\\ \sum -2x_{in}Y_i +2\sum \beta_j \sum x_{in} x_{ij}+ 2\lambda \beta_n\end{pmatrix} \\ &= A\beta +r \end{align} Where \begin{align} A = 2\begin{pmatrix} \lambda +\sum x_{i1}x_{i1} & \sum x_{i1}x_{i2} & \sum x_{i1}x_{i3} & ... & \sum x_{i1}x_{in} \\ \sum x_{i2}x_{i1} & \lambda + \sum x_{i2}x_{i2} & \sum x_{i2}x_{i3} & ... & \sum x_{i2}x_{in} \\ ... & ... & ... & ... & ... & \\ \sum x_{in}x_{i1} & \sum x_{in}x_{i2} & \sum x_{in}x_{i3} & ... &\lambda+ \sum x_{in}x_{in} \end{pmatrix} \end{align} And \begin{align} r = -2\begin{pmatrix}\sum x_{i1}Y_i \\ ..\\\sum x_{in}Y_i \end{pmatrix} \end{align} To obtain a minima, we need to solve $A\beta+r=0$. This gives you $\beta$.