0
$\begingroup$

Compute the following derivative (in matrix form) $$\frac{\partial\, \|Ax\|_2}{\partial x}$$ where $A$ is an arbitrary matrix and $x$ is a vector.

I think somebody said that the result is $2A^TAx$, but I cannot get even there. I have no idea how to develop this norm and derive it in matrix form. Because if it were $$\frac{\partial \|x\|}{\partial x}=\frac{\partial x^Tx}{\partial x}=x$$ but with that matrix in the middle. I do not know how to solve that. Thanks in advance.

$\endgroup$
  • 1
    $\begingroup$ The gradient of the square of the Euclidean norm of $Ax$, $\| Ax \|_{2}^{2}$, is $2A^{T}Ax$. $\endgroup$ – Brian Borchers Mar 30 at 13:45
  • $\begingroup$ How do you define $\partial/(\partial x)$ where $x$ is a vector? $\endgroup$ – user370967 Mar 30 at 13:48
  • $\begingroup$ @BrianBorchers how do you derive to that result $2A^TAx$? $\endgroup$ – alienflow Mar 30 at 13:51
  • 1
    $\begingroup$ Compute the directional derivative. Then extract the gradient. Take a look at this. $\endgroup$ – Rodrigo de Azevedo Mar 30 at 14:01
  • 1
    $\begingroup$ Or alternatively take a look at this: math.stackexchange.com/a/2890663/550103 $\endgroup$ – user550103 Mar 30 at 14:14
3
$\begingroup$

In the Euclidean norm ($p$ norm for $p=2$) $$ \lVert y \rVert = \left( \sum_i y_i^2 \right)^{1/2} $$ for the $k$–th coordinate of the gradient by applying the chain rule several times we have $$ \begin{align} \partial_k \lVert Ax \rVert &= \frac{\partial}{\partial x_k} \left( \sum_i \left( \sum_j a_{ij}x_j \right)^2 \right)^{1/2} \\ &= \frac{1}{2 \lVert Ax\rVert} \sum_i 2 \left( \sum_j a_{ij}x_j \right) a_{ij} \delta_{jk} \\ &= \frac{1}{\lVert Ax\rVert} \sum_i a_{ik} \left( \sum_j a_{ij}x_j \right) \\ &= \frac{1}{\lVert Ax\rVert}\left( A^T A x \right)_k \end{align} $$ Note:Funny enough here I was able to recycle another answer from this morning.

$\endgroup$
4
$\begingroup$

It is useful to introduce the Frobenius inner product as:

$$ A:B = \operatorname{tr}(A^TB)$$

with the following properties derivied from the underlying trace function

$$\eqalign{A:BC &= B^TA:C\cr &= AC^T:B\cr &= A^T:(BC)^T\cr &= BC:A \cr } $$

Then we work with differentials to find the gradient. Your problem becomes, with $u=Ax$

$$\eqalign{ f &= \|u\|_{F}^{2} = u:u \\ df &= 2u : du\\ &= 2Ax : A dx\\ &= 2A^TAx : dx} $$

Thus

$$ \frac{\partial f}{\partial x} = 2 A^TAx$$

EDIT:

For $g = \|u\|_{2} $ this becomes:

$$\eqalign{ g&= f^{1/2} \\ dg &= \frac{1}{2} f^{-\frac{1}{2}} : df\\ dg &= \frac{1}{2} f^{-\frac{1}{2}} : 2u : du\\ &=f^{-\frac{1}{2}} u : du\\ &=\frac{1}{||Ax||} Ax : Adx\\ &=\frac{1}{||Ax||} A^TAx : dx\\ }$$

$$\frac{\partial g}{\partial x}= \frac{1}{||Ax||}A^T A x$$

$\endgroup$
  • $\begingroup$ Nice answer! Alienflow also asked for $\frac{\partial g}{\partial x}$ where $g(x) = ||A x ||$. Could you add $\frac{\partial g}{\partial x}= \frac1{||Ax||}A^T A x$ to your answer? $\endgroup$ – irchans Mar 30 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.