# Gradient And Hessian Of General 2-Norm

Given $f(\mathbf{x}) = \|\mathbf{Ax}\|_2 = (\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{1/2}$,

$\nabla f(\mathbf{x}) = \frac {\mathbf{A}^\mathrm{T} \mathbf{Ax}} {\|\mathbf{Ax}\|_2} = \frac {\mathbf{A}^\mathrm{T} \mathbf{Ax}} {(\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{1/2}}$

$\nabla^2 f(\mathbf{x}) = \frac { (\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{1/2} \cdot \mathbf{A}^\mathrm{T} \mathbf{A} - (\mathbf{A}^\mathrm{T} \mathbf{Ax})^\mathrm{T} (\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{-1/2} \mathbf{A}^\mathrm{T} \mathbf{Ax} } {(\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} ) } = \frac { \mathbf{A}^\mathrm{T} \mathbf{A} } { (\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{1/2}} - \frac {\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{A} \mathbf{A}^\mathrm{T} \mathbf{Ax} } { (\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{3/2} }$

I guess I am looking for confirmation that I have done the above correctly. The dimensions match up except for the second term of the Hessian is a scalar, which makes me think that something is missing.

Edit: Also, the last equality reduces to

$\nabla^2 f(\mathbf{x}) = \frac {\mathbf{A}^\mathrm{T} \mathbf{A} - \nabla f(\mathbf{x})^\mathrm{T} \nabla f(\mathbf{x})} {\|\mathbf{Ax}\|_2}$

-
My advice is to do it first without the square root. Given your $f,$ define $g = f^2$ and find the gradient and Hessian. Just to confirm your suspicions, the gradient really is a vector, the Hessian really is a square (symmetric) matrix. –  Will Jagy Jul 25 '12 at 23:26
Try squaring and expanding $f^2(x+\delta)$. The gradient and Hessian (of $f^2$) should be fairly clear. –  copper.hat Jul 25 '12 at 23:27
Also, do this in agonizing detail for the case of just two variables, you could call them $x_1,x_2$ if you like, but you could also stick with $x,y.$ The gradient becomes a vector with just two functions as entries, the Hessian a 2 by 2 matrix. You are getting a bit lost in the formalism. –  Will Jagy Jul 25 '12 at 23:38
It looks to me like everything is fine except that the numerator should be $A^TA - (\nabla f)(\nabla f)^T$. (Also, $\nabla^2$ usually denotes the Laplacian, which is the trace of the Hessian.) –  Rahul Jul 26 '12 at 4:46

It is easier to work with $\phi(x) = \frac{1}{2} f^2(x)$. Just expand $\phi$ around $x$.

$\phi(x+\delta) = \frac{1}{2} (x + \delta)^T A^T A (x + \delta) = \phi(x) + x^TA^TA \delta + \frac{1}{2} \delta^T A^T A \delta$. It follows from this that the gradient $\nabla \phi(x) = A^T A x$, and the Hessian is $H = A^TA$.

To finish, let $g(x) = \sqrt{2x}$, and note that $f = g \circ \phi$. To get the first derivative, use the composition rule to get $D f(x) = Dg(\phi(x)) D \phi(x)$, which gives $Df(x) = \frac{1}{\sqrt{2 \phi(x)}} x^T A^T A = \frac{1}{\|Ax\|} x^T A^T A$.

Let $\eta(x) = \frac{1}{\|Ax\|}$, and $\gamma(x) = x^T A^T A$, and note that $D f(x) = \eta(x) \cdot \gamma(x)$, so we can use the product rule. Let $h(x) = Df(x)$ then the product rule gives $D h(x) (\delta) = (D \eta(x) (\delta)) \gamma(x) + \eta(x) D \gamma(x) (\delta)$.

Expanding this yields: $Dh(x)(\delta) = (- \frac{1}{\|Ax\|^2} \frac{1}{\|Ax\|} x^T A^T A \delta) x^T A^T A + \frac{1}{\|Ax\|} \delta^T A^T A$. Noting that $x^T A^T A \delta = \delta^T A^T A x$, we can write this as: $$Dh(x)(\delta) = \delta^T(\frac{1}{\|Ax\|} A^T A - \frac{1}{\|Ax\|^3 } A^T A x x^T A^T A),$$ or alternatively: $$D^2 f(x) = \frac{1}{\|Ax\|} A^T A - \frac{1}{\|Ax\|^3 } A^T A x x^T A^T A .$$

The only difference with the formula given in the question is that the latter dyad was written incorrectly (instead of the dyad $g g^T$, you have the scalar $g^T g$).

-
This is helpful but not complete. To get to the answer, one now needs the chain rule to find the gradient and Hessian of $f(\mathbf{x})=2\sqrt{\phi(\mathbf{x})}$. –  AnonSubmitter85 Jul 26 '12 at 2:10
@AnonSubmitter85: I elaborated my answer a little more. –  copper.hat Jul 26 '12 at 5:36
Thank you. This has helped me tremendously. –  AnonSubmitter85 Jul 26 '12 at 18:18
You're very welcome. –  copper.hat Jul 26 '12 at 18:30