Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I want to solve the following equation $$\frac{\partial}{\partial {\bf \beta}} \left[||{\bf y}-{\bf X}{\bf \beta}||^2 + ||{\bf \beta}||^2\right] = 0$$ for $\beta$. Here ${\bf y}$ and ${\bf \beta}$ are vectors and ${\bf X}$ is a matrix. I am having trouble with the part of differentiating the equation. I can split it up into $$\frac{\partial}{\partial {\bf \beta}} ||{\bf y}-{\bf X}{\bf \beta}||^2 + \frac{\partial}{\partial {\bf \beta}}||{\bf \beta}||^2$$ and then use the rule that $$\frac{\partial}{\partial a}||a||^2 = 2a$$

The problem is with the other part. I can use the product rule, but I am still left with $\frac{\partial}{\partial {\bf \beta}}||{\bf y} - {\bf X}{\bf \beta}||^2$.

share|improve this question
What norm do you use? $||A||^2= tr(A^t A)$? If so, just use linearity of the trace functional and the product rule. –  Sam Feb 6 '13 at 23:16
I am using the 2-norm. –  user61300 Feb 6 '13 at 23:19
Your question says matrix norms but it seems like only $X$ is a matrix and $y$ and $\beta$ are vectors. Is that correct? –  Dominique Feb 7 '13 at 1:26
You are actually correct, I will change the question to reflect this. –  user61300 Feb 7 '13 at 1:56

2 Answers 2

up vote 1 down vote accepted

$$ \frac{\partial}{\partial \beta} \left(\|F(\beta)\|^2\right) = \frac{\partial}{\partial \beta} \left(F(\beta) \cdot F(\beta)\right) = 2 \left( \frac{\partial}{\partial \beta} F(\beta) \right) \cdot F(\beta) $$

share|improve this answer
How would I get rid of the $\frac{\partial}{\partial} F(\beta) = \frac{\partial}{\partial} ||y-X\beta||$? –  user61300 Feb 6 '13 at 23:20
No, take $F(\beta) = y - X \beta$. $\dfrac{\partial}{\partial \beta} (y - X \beta) = - X^T$ (i.e. $\dfrac{\partial}{\partial \beta_i} (y - X \beta)_j = - X_{ji}$). –  Robert Israel Feb 7 '13 at 2:28

Let's do a directional derivative instead, eventually building up to some voodoo magic.

$$a \cdot \nabla_\beta [(y - \underline X(\beta))^2 + \beta^2] = -\underline X(a) \cdot [-2(y - \underline X(\beta))] + 2 \beta \cdot a$$

But $\underline X(a) \cdot b = \overline X(b) \cdot a$. This exchanges a linear operator with its adjoint.

We can then use this to write the result as

$$2a \cdot [\overline X(\underline X(\beta)-y) + \beta]$$

Now we can take out the $a$ to get

$$2[\overline X(\underline X(\beta)-y) + \beta]$$

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.