54
$\begingroup$

Could someone please provide a proof for the following rule:

$$\nabla\|x\|_2^2 = 2x$$

I.E. why is the gradient of the $L_2$ norm square of $x$ equal to $2x$?

Thanks

$\endgroup$
0
74
$\begingroup$

Use the definition. If $$f(x)=\|x\|^2_2= \left(\left(\sum_{k=1}^n x_k^2 \right)^{1/2}\right)^{2}=\sum_{k=1}^n x_k^2 ,$$ then $$\frac{\partial}{\partial x_j}f(x) =\frac{\partial}{\partial x_j}\sum_{k=1}^n x_k^2=\sum_{k=1}^n \underbrace{\frac{\partial}{\partial x_j}x_k^2}_{\substack{=0, \ \text{ if } j \neq k,\\=2x_j, \ \text{ else }}}= 2x_j.$$ It follows that $$\nabla f(x) = 2x.$$

$\endgroup$
6
  • $\begingroup$ Thanks. One follow up question, does this still hold if x is complex? $\endgroup$ – user167133 Jul 30 '14 at 20:56
  • $\begingroup$ Isn't the L2 norm defined as: $$f(x)=\|x\|^2_2= \left(\left(\sum_{k=1}^n |x_k|^2 \right)^{1/2}\right)^{2}=\sum_{k=1}^n |x_k|^2$$, there is an absolute value sign surrounding $x_k$, so when you take the derivative what should pop out is $2|x_j|$ not just $2x_j$, correct me if I am wrong $\endgroup$ – Rodrigo Amarante Oct 23 '15 at 17:46
  • 2
    $\begingroup$ @Lookbehindyou well |t|^2 = t^2 for any $t\in\Bbb R$. Moreover, note that $\frac{d}{t}|t|^2 = 2|t|\operatorname{sign}(t)=2t$, where $\operatorname{sign}$ is the sign function $\endgroup$ – Surb Feb 29 '16 at 14:49
  • $\begingroup$ Could you please explain what happened to the sigma in the last step? $\endgroup$ – Gigili Aug 16 '16 at 12:38
  • $\begingroup$ @Gigili: If $k\neq j$, then $\frac{\partial }{\partial x_j}x_k^2=0$. Then, $$\sum_{k=1}^n\frac{\partial }{\partial x_j}x_k^2=0+...+0+2x_j+0+...+0=2x_j.$$ $\endgroup$ – Surb Aug 16 '16 at 12:44
30
$\begingroup$

Another approach that extends to more general settings is to use the connection between the norm and the inner product, $$\|x\|^2 = (x,x).$$

We have the finite difference, \begin{align} \|x+sh\|^2 - \|x\|^2 &= (x+sh,x+sh) - (x,x) \\ &= (x,x) + 2s(x,h) + s^2(h,h) - (x,x) \\ &= 2s(x,h) + s^2(h,h). \end{align}

The gradient acting in the direction $h$ is the limit of this finite difference as the stepsize goes to zero, \begin{align} (\nabla\|x\|^2, h) &:= \lim_{s \rightarrow 0} \frac{1}{s}\left[\|x+sh\|^2 - \|x\|^2\right] \\ &= \lim_{s \rightarrow 0} \frac{1}{s}\left[2s(x,h) + s^2(h,h)\right] \\ &= (2x,h). \end{align} Since this holds for any direction $h$, the gradient must be $\nabla \|x\|^2 = 2x$.

$\endgroup$
1
$\begingroup$

I'm not sure if this is rigorous enough to count as a proof, but an elegant way to obtain derivatives of vector expressions is to use matrix differential calculus.

Let $y = \lVert x \rVert_2^2 = x^{T} x$ with $x \in \mathbb{R}^{n}$. Using the product rule, the differential of $y$ is $$ dy = dx^{T} x + x^{T} dx = 2 x^{T} dx $$

We can then set $$ dy = \frac{dy}{dx} dx = (\nabla_{x} y)^{T} dx = 2x^{T} dx $$ where $dy/dx \in \mathbb{R}^{1 \times n}$ is called the derivative (a linear operator) and $\nabla_{x} y \in \mathbb{R}^{n}$ is called the gradient (a vector).

Now we can see $\nabla_{x} y = 2 x$.


If $x$ is complex, the complex derivative does not exist because $z \mapsto |z|^{2}$ is not a holomorphic function.

We can, however, instead consider the real derivatives with respect to the two components of $x$. Let $x = u + i v$. With this definition, $y$ is a real function of $u, v \in \mathbb{R}^{n}$ defined by $$ y = x^* x = (u + i v)^* (u + i v) = u^T u - v^T v $$ Taking the differential $$ dy = 2 u^T du - 2 v^T dv = \frac{\partial y}{\partial u} du + \frac{\partial y}{\partial v} dv $$ and therefore $$ \nabla_{u} y = 2 u \enspace , \qquad \nabla_{v} y = -2 v $$


For an introduction to matrix differential calculus, see the lecture of Geoff Gordon on YouTube or the paper on matrix derivatives of Mike Giles.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.