Gradient and Hessian of this function What is the gradient $\nabla_x$ / hessian $\nabla_x^2$ of $\frac{(Ax)^{\top}y}{||Ax||}$ where $A \in \mathbb{R}^{m \, \times \, n}$, $x \in \mathbb{R}^n$ and $y \in \mathbb{R}^m$?
I don't really know how to approach this problem because I don't know how to deal with quotients in matrix calculus.
 A: For convenience, let's define some variables
$$\eqalign{
b &= Ax &\implies &db = A\,dx \cr
1 &= \alpha\beta &\implies &d\alpha = -\alpha^2d\beta \cr
\beta^2 &= b^Tb &\implies &\beta\,d\beta = b^Tdb=b^TA\,dx \cr
 &\, &\, &d\beta = \alpha\,b^Tdb=\alpha\,b^TA\,dx = z^TA\,dx \cr
z &= \alpha b &\implies &dz =\alpha\,db-b\alpha^3(b^Tdb)=\alpha(I-zz^T)A\,dx \cr
\cr}$$
Write the function in terms of these variables and find its differential and gradient
$$\eqalign{
f &= y^Tz \cr\cr
df &= y^Tdz \cr
 &= y^T\alpha(I-zz^T)A\,dx \cr
   &= \alpha(y^T-fz^T)A\,dx \cr
\cr
g^T=\frac{\partial f}{\partial x^T} &= \alpha(y-fz)^TA \cr\cr
}$$
Now find the differential and gradient of the gradient
$$\eqalign{
dg^T &= d\alpha(y-fz)^TA -df\,\alpha z^TA -\alpha f\,dz^TA \cr
&= -\alpha\,d\beta\,g^T -\alpha(dx^Tg)z^TA -\alpha^2f\,dx^TA^T(I-zz^T)A \cr
&= -\alpha\,dx^T\Big(A^Tzg^T +gz^TA +\alpha f(A^TA-A^Tzz^TA)\Big) \cr\cr
H = \frac{\partial g^T}{\partial x}
&= -\alpha\Big(A^Tzg^T +gz^TA +\alpha f(A^TA-A^Tzz^TA)\Big) \cr
\cr}$$
This can be made nicer by introducing yet more variables
$$\eqalign{
 M &= \alpha A,\,\,\,\,\,\, p = M^Tz \cr
}$$ yielding
$$\eqalign{
 g &= M^Ty-fp \cr
 H &= fpp^T - pg^T - gp^T - fM^TM \cr
}$$
A: Strategy to approach problem:


*

*Let $g(x) = (Ax)^Ty$ and $h(x) = \|Ax\|$, then the function you want to differentiate is $f(x) = g(x)h(x)^{-1}$.

*Recall that $\nabla_x = (\partial_1, \partial_2, \dots \partial_n)$ where we use the shorthand $\partial_i = \partial/\partial x^i$, so it suffices to compute $\partial_if = (\partial_i g) h^{-1} + g\partial_i(h^{-1}) =  (\partial_i g) h^{-1} -g(\partial_ih)/h^2$.

*It remains to compute $\partial_i g$ and $\partial_i h$.  This is easiest if one writes these functions in index notation (especially if one uses the Einstein summation notation to avoid summation symbols).  For example
$$
  \partial_i g = \partial_i[(Ax)^jy_j] = \partial_i(A^j_kx^ky_j) = A^i_ky_j\partial_ix^k=A^j_ky_j\delta^k_i = A^j_iy_j
$$


The Hessian is more work since in index notation is a matrix of mixed partial derivatives, but essentially the same strategy works for computing an arbitrary matrix entry.
