# Can Gauss-Newton algorithm give better optimization performances than Newton algorithm?

I am currently facing the problem of a robotic manipulator calibration: the goal is to find the best correction that must be applied to a set of kinematic parameters describing the robot model, in order to minimize the position error in each different configuration of joint angles.

The objective function is the classical sum of squares $$f(\boldsymbol k) = \sum_{i=1}^{N}\|\boldsymbol e_i(\boldsymbol k)\|^2 = \sum_{i=1}^{N}\|{\overline {\boldsymbol r}}_i - {\boldsymbol r}_i(\boldsymbol k)\|^2 = \sum_{i=1}^{N} f_i(\boldsymbol k)$$

$$f_i(\boldsymbol k) = \|\boldsymbol e_i(\boldsymbol k)\|^2 = \|{\overline {\boldsymbol r}}_i - {\boldsymbol r}_i(\boldsymbol k)\|^2$$ where ${\overline {\boldsymbol r}}_i = (\overline x_i, \overline y_i, \overline z_i)$ is the measured cartesian position in $i$-th configuration while ${\boldsymbol r}_i(\boldsymbol k) = (x_i(\boldsymbol k),y_i(\boldsymbol k),z_i(\boldsymbol k))$ is the cartesian position obtained from model using $\boldsymbol k$ as parameter vector which is also the decision variable.

In order to perform minimization using Newton method the gradient and the hessian matrix of the objective function are needed: $$\nabla f(\boldsymbol k) = \sum_{i=1}^{N}\nabla f_i(\boldsymbol k)$$ $$\nabla f_i(\boldsymbol k) = -2 \left(\nabla x_i(\boldsymbol k) \Delta x_i(\boldsymbol k)+ \nabla y_i(\boldsymbol k) \Delta y_i(\boldsymbol k) + \nabla z_i(\boldsymbol k) \Delta z_i(\boldsymbol k) \right) = -2 \left( \boldsymbol J \boldsymbol r_i(\boldsymbol k)\right)^T\boldsymbol e_i(\boldsymbol k)$$ where ${\boldsymbol e}_i(\boldsymbol k) = (\Delta x_i(\boldsymbol k),\Delta y_i(\boldsymbol k), \Delta z_i(\boldsymbol k))$ and $$\boldsymbol J \boldsymbol r_i(\boldsymbol k) =\begin{pmatrix} \nabla x_i^T(\boldsymbol k) \\ \nabla y_i^T(\boldsymbol k)\\ \nabla z_i^T(\boldsymbol k)\end{pmatrix}$$ is the position jacobian matrix. $$\boldsymbol Hf(\boldsymbol k) = \sum_{i=1}^{N} \boldsymbol Hf_i(\boldsymbol k)$$ $$\boldsymbol Hf_i(\boldsymbol k) = \boldsymbol J\nabla f_i(\boldsymbol k) = -2 \left(- \nabla x_i(\boldsymbol k) \nabla x_i^T(\boldsymbol k) - \nabla y_i(\boldsymbol k) \nabla y_i^T(\boldsymbol k) - \nabla z_i(\boldsymbol k) \nabla z_i^T(\boldsymbol k) + \Delta x_i(\boldsymbol k) \boldsymbol Hx_i(\boldsymbol k)+ \Delta y_i(\boldsymbol k) \boldsymbol Hy_i(\boldsymbol k) + \Delta z_i(\boldsymbol k) \boldsymbol Hz_i(\boldsymbol k) \right) = 2 \left( \boldsymbol J \boldsymbol r_i^T(\boldsymbol k) \boldsymbol J \boldsymbol r_i(\boldsymbol k) - \Delta x_i(\boldsymbol k) \boldsymbol Hx_i(\boldsymbol k)- \Delta y_i(\boldsymbol k) \boldsymbol Hy_i(\boldsymbol k) - \Delta z_i(\boldsymbol k) \boldsymbol Hz_i(\boldsymbol k) \right)$$

So the objective function can be locally approximated as a quadratic function: $$f(\boldsymbol k + \Delta \boldsymbol k ) \approx f(\boldsymbol k) + \nabla f(\boldsymbol k)^T \Delta \boldsymbol k + \frac{1}{2} \Delta \boldsymbol k^T \boldsymbol Hf(\boldsymbol k) \Delta \boldsymbol k$$ its minimum lies where the gradient vanishes: $$\nabla f(\boldsymbol k+\Delta \boldsymbol k) \approx \nabla f(\boldsymbol k)+ \boldsymbol Hf(\boldsymbol k) \Delta \boldsymbol k = \boldsymbol 0 \Rightarrow$$ $$\boldsymbol Hf(\boldsymbol k) \Delta \boldsymbol k^{*} = -\nabla f(\boldsymbol k)$$ Solving iteratively this equation and updating the decision variable constitutes the Newton method. It requires the knowledge or computation of both first and second derivative of all components of the error vector.

On the contrary, since the residues $\boldsymbol e_i = \Delta \boldsymbol r_i$ are generally small, one can approximated the hessian matrix by neglecting all the second derivatives: $$\boldsymbol Hf_i(\boldsymbol k) \approx 2 \boldsymbol J \boldsymbol r_i^T(\boldsymbol k) \boldsymbol J \boldsymbol r_i(\boldsymbol k)$$ and in this case the iterative solution is known as Gauss-Newton. In both cases, the model is minimal then the jacobian matrix is full-rank hence the hessian matrix can be directly inverted.

My question is: in presence of exact first and second derivatives, is it possible that Gauss-Newton method (which uses only a part, the prevalent one, of the hessian matrix) converges to the solution in a better way (with less iterations or with lower value for objective function and gradient norm at equal iterations) than the Newton method (which exploits the full hessian matrix, very different when the residues are not negligible)?

And if yes, which is the reason of this inexplicable behaviour (sometimes, if the initial guess is not so close, newton method seems to fail whereas gauss-newton converges nonethelss).