Could someone please provide a proof for why the gradient of the squared $$2$$-norm of $$x$$ is equal to $$2x$$?

$$\nabla\|x\|_2^2 = 2x$$

• Related Mar 15 at 9:20

Use the definition. If $$f(x)=\|x\|^2_2= \left(\left(\sum_{k=1}^n x_k^2 \right)^{1/2}\right)^{2}=\sum_{k=1}^n x_k^2 ,$$ then $$\frac{\partial}{\partial x_j}f(x) =\frac{\partial}{\partial x_j}\sum_{k=1}^n x_k^2=\sum_{k=1}^n \underbrace{\frac{\partial}{\partial x_j}x_k^2}_{\substack{=0, \ \text{ if } j \neq k,\\=2x_j, \ \text{ else }}}= 2x_j.$$ It follows that $$\nabla f(x) = 2x.$$

• Thanks. One follow up question, does this still hold if x is complex? Jul 30, 2014 at 20:56
• Isn't the L2 norm defined as: $$f(x)=\|x\|^2_2= \left(\left(\sum_{k=1}^n |x_k|^2 \right)^{1/2}\right)^{2}=\sum_{k=1}^n |x_k|^2$$, there is an absolute value sign surrounding $x_k$, so when you take the derivative what should pop out is $2|x_j|$ not just $2x_j$, correct me if I am wrong Oct 23, 2015 at 17:46
• @Lookbehindyou well |t|^2 = t^2 for any $t\in\Bbb R$. Moreover, note that $\frac{d}{t}|t|^2 = 2|t|\operatorname{sign}(t)=2t$, where $\operatorname{sign}$ is the sign function
– Surb
Feb 29, 2016 at 14:49
• Could you please explain what happened to the sigma in the last step? Aug 16, 2016 at 12:38
• @Gigili: If $k\neq j$, then $\frac{\partial }{\partial x_j}x_k^2=0$. Then, $$\sum_{k=1}^n\frac{\partial }{\partial x_j}x_k^2=0+...+0+2x_j+0+...+0=2x_j.$$
– Surb
Aug 16, 2016 at 12:44

Another approach that extends to more general settings is to use the connection between the norm and the inner product, $$\|x\|^2 = (x,x).$$

We have the finite difference, \begin{align} \|x+sh\|^2 - \|x\|^2 &= (x+sh,x+sh) - (x,x) \\ &= (x,x) + 2s(x,h) + s^2(h,h) - (x,x) \\ &= 2s(x,h) + s^2(h,h). \end{align}

The gradient acting in the direction $$h$$ is the limit of this finite difference as the stepsize goes to zero, \begin{align} (\nabla\|x\|^2, h) &:= \lim_{s \rightarrow 0} \frac{1}{s}\left[\|x+sh\|^2 - \|x\|^2\right] \\ &= \lim_{s \rightarrow 0} \frac{1}{s}\left[2s(x,h) + s^2(h,h)\right] \\ &= (2x,h). \end{align} Since this holds for any direction $$h$$, the gradient must be $$\nabla \|x\|^2 = 2x$$.

I'm not sure if this is rigorous enough to count as a proof, but an elegant way to obtain derivatives of vector expressions is to use matrix differential calculus.

Let $$y = \lVert x \rVert_2^2 = x^{T} x$$ with $$x \in \mathbb{R}^{n}$$. Using the product rule, the differential of $$y$$ is $$dy = dx^{T} x + x^{T} dx = 2 x^{T} dx$$

We can then set $$dy = \frac{dy}{dx} dx = (\nabla_{x} y)^{T} dx = 2x^{T} dx$$ where $$dy/dx \in \mathbb{R}^{1 \times n}$$ is called the derivative (a linear operator) and $$\nabla_{x} y \in \mathbb{R}^{n}$$ is called the gradient (a vector).

Now we can see $$\nabla_{x} y = 2 x$$.

If $$x$$ is complex, the complex derivative does not exist because $$z \mapsto |z|^{2}$$ is not a holomorphic function.

We can, however, instead consider the real derivatives with respect to the two components of $$x$$. Let $$x = u + i v$$. With this definition, $$y$$ is a real function of $$u, v \in \mathbb{R}^{n}$$ defined by $$y = x^* x = (u + i v)^* (u + i v) = u^T u - v^T v$$ Taking the differential $$dy = 2 u^T du - 2 v^T dv = \frac{\partial y}{\partial u} du + \frac{\partial y}{\partial v} dv$$ and therefore $$\nabla_{u} y = 2 u \enspace , \qquad \nabla_{v} y = -2 v$$

For an introduction to matrix differential calculus, see the lecture of Geoff Gordon on YouTube or the paper on matrix derivatives of Mike Giles.

• sorry me, why $x^T dx + dx^T x = 2x^T dx$ ? I suppose, that dx^T = dx, because it is number, but what we can do with x + x^T .. Nov 7, 2021 at 23:13
• Actually $dx$ has the same dimension as $x$, both are in $R^n$ Nov 9, 2021 at 7:56
• In case it wasn't clear - for $a, b \in R^n$, we have $a^T b = b^T a = \sum_i a_i b_i$ May 3 at 8:14