# Differentiation of vector norms

I want to solve the following equation $$\frac{\partial}{\partial {\bf \beta}} \left[||{\bf y}-{\bf X}{\bf \beta}||^2 + ||{\bf \beta}||^2\right] = 0$$ for $\beta$. Here ${\bf y}$ and ${\bf \beta}$ are vectors and ${\bf X}$ is a matrix. I am having trouble with the part of differentiating the equation. I can split it up into $$\frac{\partial}{\partial {\bf \beta}} ||{\bf y}-{\bf X}{\bf \beta}||^2 + \frac{\partial}{\partial {\bf \beta}}||{\bf \beta}||^2$$ and then use the rule that $$\frac{\partial}{\partial a}||a||^2 = 2a$$

The problem is with the other part. I can use the product rule, but I am still left with $\frac{\partial}{\partial {\bf \beta}}||{\bf y} - {\bf X}{\bf \beta}||^2$.

-
What norm do you use? $||A||^2= tr(A^t A)$? If so, just use linearity of the trace functional and the product rule. – Sam Feb 6 '13 at 23:16
I am using the 2-norm. – user61300 Feb 6 '13 at 23:19
Your question says matrix norms but it seems like only $X$ is a matrix and $y$ and $\beta$ are vectors. Is that correct? – Dominique Feb 7 '13 at 1:26
You are actually correct, I will change the question to reflect this. – user61300 Feb 7 '13 at 1:56

$$\frac{\partial}{\partial \beta} \left(\|F(\beta)\|^2\right) = \frac{\partial}{\partial \beta} \left(F(\beta) \cdot F(\beta)\right) = 2 \left( \frac{\partial}{\partial \beta} F(\beta) \right) \cdot F(\beta)$$ $F(\beta) \in \mathcal{R}^D$, where $D$ is the dimension of $F(\beta)$.

-
How would I get rid of the $\frac{\partial}{\partial} F(\beta) = \frac{\partial}{\partial} ||y-X\beta||$? – user61300 Feb 6 '13 at 23:20
No, take $F(\beta) = y - X \beta$. $\dfrac{\partial}{\partial \beta} (y - X \beta) = - X^T$ (i.e. $\dfrac{\partial}{\partial \beta_i} (y - X \beta)_j = - X_{ji}$). – Robert Israel Feb 7 '13 at 2:28
What would you do if you had a cubed norm instead? – Dr. Johnny Mohawk Dec 31 '15 at 14:15

Let's do a directional derivative instead, eventually building up to some voodoo magic.

$$a \cdot \nabla_\beta [(y - \underline X(\beta))^2 + \beta^2] = -\underline X(a) \cdot [-2(y - \underline X(\beta))] + 2 \beta \cdot a$$

But $\underline X(a) \cdot b = \overline X(b) \cdot a$. This exchanges a linear operator with its adjoint.

We can then use this to write the result as

$$2a \cdot [\overline X(\underline X(\beta)-y) + \beta]$$

Now we can take out the $a$ to get

$$2[\overline X(\underline X(\beta)-y) + \beta]$$

-