Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want to solve the following equation $$\frac{\partial}{\partial {\bf \beta}} \left[||{\bf y}-{\bf X}{\bf \beta}||^2 + ||{\bf \beta}||^2\right] = 0$$ for $\beta$. Here ${\bf y}$ and ${\bf \beta}$ are vectors and ${\bf X}$ is a matrix. I am having trouble with the part of differentiating the equation. I can split it up into $$\frac{\partial}{\partial {\bf \beta}} ||{\bf y}-{\bf X}{\bf \beta}||^2 + \frac{\partial}{\partial {\bf \beta}}||{\bf \beta}||^2$$ and then use the rule that $$\frac{\partial}{\partial a}||a||^2 = 2a$$

The problem is with the other part. I can use the product rule, but I am still left with $\frac{\partial}{\partial {\bf \beta}}||{\bf y} - {\bf X}{\bf \beta}||^2$.

share|cite|improve this question
What norm do you use? $||A||^2= tr(A^t A)$? If so, just use linearity of the trace functional and the product rule. – Sam Feb 6 '13 at 23:16
I am using the 2-norm. – user61300 Feb 6 '13 at 23:19
Your question says matrix norms but it seems like only $X$ is a matrix and $y$ and $\beta$ are vectors. Is that correct? – Dominique Feb 7 '13 at 1:26
You are actually correct, I will change the question to reflect this. – user61300 Feb 7 '13 at 1:56
up vote 3 down vote accepted

$$ \frac{\partial}{\partial \beta} \left(\|F(\beta)\|^2\right) = \frac{\partial}{\partial \beta} \left(F(\beta) \cdot F(\beta)\right) = 2 \left( \frac{\partial}{\partial \beta} F(\beta) \right) \cdot F(\beta) $$ $F(\beta) \in \mathcal{R}^D$, where $D$ is the dimension of $F(\beta)$.

share|cite|improve this answer
How would I get rid of the $\frac{\partial}{\partial} F(\beta) = \frac{\partial}{\partial} ||y-X\beta||$? – user61300 Feb 6 '13 at 23:20
No, take $F(\beta) = y - X \beta$. $\dfrac{\partial}{\partial \beta} (y - X \beta) = - X^T$ (i.e. $\dfrac{\partial}{\partial \beta_i} (y - X \beta)_j = - X_{ji}$). – Robert Israel Feb 7 '13 at 2:28
What would you do if you had a cubed norm instead? – Dr. Johnny Mohawk Dec 31 '15 at 14:15

Let's do a directional derivative instead, eventually building up to some voodoo magic.

$$a \cdot \nabla_\beta [(y - \underline X(\beta))^2 + \beta^2] = -\underline X(a) \cdot [-2(y - \underline X(\beta))] + 2 \beta \cdot a$$

But $\underline X(a) \cdot b = \overline X(b) \cdot a$. This exchanges a linear operator with its adjoint.

We can then use this to write the result as

$$2a \cdot [\overline X(\underline X(\beta)-y) + \beta]$$

Now we can take out the $a$ to get

$$2[\overline X(\underline X(\beta)-y) + \beta]$$

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.