Given $f(\mathbf{x}) = \|\mathbf{Ax}\|_2 = (\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{1/2}$,

$\nabla f(\mathbf{x}) = \frac {\mathbf{A}^\mathrm{T} \mathbf{Ax}} {\|\mathbf{Ax}\|_2} = \frac {\mathbf{A}^\mathrm{T} \mathbf{Ax}} {(\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{1/2}}$

$\nabla^2 f(\mathbf{x}) = \frac { (\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{1/2} \cdot \mathbf{A}^\mathrm{T} \mathbf{A} - (\mathbf{A}^\mathrm{T} \mathbf{Ax})^\mathrm{T} (\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{-1/2} \mathbf{A}^\mathrm{T} \mathbf{Ax} } {(\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} ) } = \frac { \mathbf{A}^\mathrm{T} \mathbf{A} } { (\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{1/2}} - \frac {\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{A} \mathbf{A}^\mathrm{T} \mathbf{Ax} } { (\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{3/2} }$

I guess I am looking for confirmation that I have done the above correctly. The dimensions match up except for the second term of the Hessian is a scalar, which makes me think that something is missing.

Edit: Also, the last equality reduces to

$\nabla^2 f(\mathbf{x}) = \frac {\mathbf{A}^\mathrm{T} \mathbf{A} - \nabla f(\mathbf{x})^\mathrm{T} \nabla f(\mathbf{x})} {\|\mathbf{Ax}\|_2}$

  • 1
    $\begingroup$ My advice is to do it first without the square root. Given your $f,$ define $g = f^2$ and find the gradient and Hessian. Just to confirm your suspicions, the gradient really is a vector, the Hessian really is a square (symmetric) matrix. $\endgroup$ – Will Jagy Jul 25 '12 at 23:26
  • 1
    $\begingroup$ Try squaring and expanding $f^2(x+\delta)$. The gradient and Hessian (of $f^2$) should be fairly clear. $\endgroup$ – copper.hat Jul 25 '12 at 23:27
  • $\begingroup$ Also, do this in agonizing detail for the case of just two variables, you could call them $x_1,x_2$ if you like, but you could also stick with $x,y.$ The gradient becomes a vector with just two functions as entries, the Hessian a 2 by 2 matrix. You are getting a bit lost in the formalism. $\endgroup$ – Will Jagy Jul 25 '12 at 23:38
  • 1
    $\begingroup$ It looks to me like everything is fine except that the numerator should be $A^TA - (\nabla f)(\nabla f)^T$. (Also, $\nabla^2$ usually denotes the Laplacian, which is the trace of the Hessian.) $\endgroup$ – Rahul Jul 26 '12 at 4:46

It is easier to work with $\phi(x) = \frac{1}{2} f^2(x)$. Just expand $\phi$ around $x$.

$\phi(x+\delta) = \frac{1}{2} (x + \delta)^T A^T A (x + \delta) = \phi(x) + x^TA^TA \delta + \frac{1}{2} \delta^T A^T A \delta$. It follows from this that the gradient $\nabla \phi(x) = A^T A x$, and the Hessian is $H = A^TA$.

To finish, let $g(x) = \sqrt{2x}$, and note that $f = g \circ \phi$. To get the first derivative, use the composition rule to get $D f(x) = Dg(\phi(x)) D \phi(x)$, which gives $Df(x) = \frac{1}{\sqrt{2 \phi(x)}} x^T A^T A = \frac{1}{\|Ax\|} x^T A^T A$.

Let $\eta(x) = \frac{1}{\|Ax\|}$, and $\gamma(x) = x^T A^T A$, and note that $D f(x) = \eta(x) \cdot \gamma(x)$, so we can use the product rule. Let $h(x) = Df(x)$ then the product rule gives $D h(x) (\delta) = (D \eta(x) (\delta)) \gamma(x) + \eta(x) D \gamma(x) (\delta)$.

Expanding this yields: $Dh(x)(\delta) = (- \frac{1}{\|Ax\|^2} \frac{1}{\|Ax\|} x^T A^T A \delta) x^T A^T A + \frac{1}{\|Ax\|} \delta^T A^T A $. Noting that $x^T A^T A \delta = \delta^T A^T A x$, we can write this as: $$Dh(x)(\delta) = \delta^T(\frac{1}{\|Ax\|} A^T A - \frac{1}{\|Ax\|^3 } A^T A x x^T A^T A),$$ or alternatively: $$D^2 f(x) = \frac{1}{\|Ax\|} A^T A - \frac{1}{\|Ax\|^3 } A^T A x x^T A^T A .$$

The only difference with the formula given in the question is that the latter dyad was written incorrectly (instead of the dyad $g g^T$, you have the scalar $g^T g$).

  • $\begingroup$ This is helpful but not complete. To get to the answer, one now needs the chain rule to find the gradient and Hessian of $f(\mathbf{x})=2\sqrt{\phi(\mathbf{x})}$. $\endgroup$ – AnonSubmitter85 Jul 26 '12 at 2:10
  • $\begingroup$ @AnonSubmitter85: I elaborated my answer a little more. $\endgroup$ – copper.hat Jul 26 '12 at 5:36
  • $\begingroup$ Thank you. This has helped me tremendously. $\endgroup$ – AnonSubmitter85 Jul 26 '12 at 18:18
  • $\begingroup$ You're very welcome. $\endgroup$ – copper.hat Jul 26 '12 at 18:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.