Computing the derivative of $\|Ax\|_2$

Compute the following derivative (in matrix form) $$\frac{\partial\, \|Ax\|_2}{\partial x}$$ where $$A$$ is an arbitrary matrix and $$x$$ is a vector.

I think somebody said that the result is $$2A^TAx$$, but I cannot get even there. I have no idea how to develop this norm and derive it in matrix form. Because if it were $$\frac{\partial \|x\|}{\partial x}=\frac{\partial x^Tx}{\partial x}=x$$ but with that matrix in the middle. I do not know how to solve that. Thanks in advance.

• The gradient of the square of the Euclidean norm of $Ax$, $\| Ax \|_{2}^{2}$, is $2A^{T}Ax$. – Brian Borchers Mar 30 at 13:45
• How do you define $\partial/(\partial x)$ where $x$ is a vector? – user370967 Mar 30 at 13:48
• @BrianBorchers how do you derive to that result $2A^TAx$? – alienflow Mar 30 at 13:51
• Compute the directional derivative. Then extract the gradient. Take a look at this. – Rodrigo de Azevedo Mar 30 at 14:01
• Or alternatively take a look at this: math.stackexchange.com/a/2890663/550103 – user550103 Mar 30 at 14:14

In the Euclidean norm ($$p$$ norm for $$p=2$$) $$\lVert y \rVert = \left( \sum_i y_i^2 \right)^{1/2}$$ for the $$k$$–th coordinate of the gradient by applying the chain rule several times we have \begin{align} \partial_k \lVert Ax \rVert &= \frac{\partial}{\partial x_k} \left( \sum_i \left( \sum_j a_{ij}x_j \right)^2 \right)^{1/2} \\ &= \frac{1}{2 \lVert Ax\rVert} \sum_i 2 \left( \sum_j a_{ij}x_j \right) a_{ij} \delta_{jk} \\ &= \frac{1}{\lVert Ax\rVert} \sum_i a_{ik} \left( \sum_j a_{ij}x_j \right) \\ &= \frac{1}{\lVert Ax\rVert}\left( A^T A x \right)_k \end{align} Note:Funny enough here I was able to recycle another answer from this morning.

It is useful to introduce the Frobenius inner product as:

$$A:B = \operatorname{tr}(A^TB)$$

with the following properties derivied from the underlying trace function

\eqalign{A:BC &= B^TA:C\cr &= AC^T:B\cr &= A^T:(BC)^T\cr &= BC:A \cr }

Then we work with differentials to find the gradient. Your problem becomes, with $$u=Ax$$

\eqalign{ f &= \|u\|_{F}^{2} = u:u \\ df &= 2u : du\\ &= 2Ax : A dx\\ &= 2A^TAx : dx}

Thus

$$\frac{\partial f}{\partial x} = 2 A^TAx$$

EDIT:

For $$g = \|u\|_{2}$$ this becomes:

\eqalign{ g&= f^{1/2} \\ dg &= \frac{1}{2} f^{-\frac{1}{2}} : df\\ dg &= \frac{1}{2} f^{-\frac{1}{2}} : 2u : du\\ &=f^{-\frac{1}{2}} u : du\\ &=\frac{1}{||Ax||} Ax : Adx\\ &=\frac{1}{||Ax||} A^TAx : dx\\ }

$$\frac{\partial g}{\partial x}= \frac{1}{||Ax||}A^T A x$$

• Nice answer! Alienflow also asked for $\frac{\partial g}{\partial x}$ where $g(x) = ||A x ||$. Could you add $\frac{\partial g}{\partial x}= \frac1{||Ax||}A^T A x$ to your answer? – irchans Mar 30 at 14:08