# Formulating the provided cost function for a least squares optimization problem

I'm trying to apply nonlinear refinement to a homography matrix using Levenberg-Marquardt optimization. I'm using what's called the "symmetric transfer error", which is defined here for a set of at least four point correspondences $x_i \leftrightarrow x_i'$.

$\sum_{i} ||x_i - H^{-1}x_i'||^2 + ||x_i' - Hx_i||^2$

To my understanding, I'm trying to find 9 parameters of the 3x3 homography matrix $H$ which minimizes the above error. In order to do this programmatically, I was planning on using scipy.optimize.least_squares but I'm having trouble formulating this as a least squares optimization problem. Mainly because it seems as though I have two distinct sets of residuals, $\{x_i - H^{-1}x_i'\} \forall i$ and $\{x_i' - Hx_i\} \forall i$.

In the documentation for the above function, I need to specify a function $f_i(x, a, b, ...)$ which returns a vector of residuals for parameter $x$ and arguments $a, b$, etc. As well as an error function $rho$ which takes in a vector and returns a scalar. I'm guessing my $rho$ is just $||f_i(x)||^2$ but please correct me if I'm wrong.

So, I guess I'm having trouble representing my loss function using a single $f$ and a single $rho$.

• It would be good to know if $H$ has ant special properties. For example, it seems that the teo components to minimise could be dealt with separately if $H$ was an isometry. – AnyAD Jun 4 '18 at 5:31
• $H$ is a homography, or a projective transform, defined up to an arbitrary scale. In other words it’s a 8DOF matrix which maps points from one plane to another – Carpetfizz Jun 4 '18 at 5:38
• I may be wrong, but it seems that you would have more than 2 terms to minimise (distinct sets of residuals), since you also have at least four points. So 8 parameters in $H$ and also $4$ of $||f(x_i)||$ and $4$ of $||H^{-1}f(x_i)||$ to minimize. – AnyAD Jun 4 '18 at 15:03
• Do you mind expanding on why? – Carpetfizz Jun 4 '18 at 15:05
• Just from reading online about the L-M method, it seems that the problem to solve needs to be in the form of minimising a $n$ vector of errors, so norm of the type $||f(x) ||^2$, where $f$ is a function from $R^m$ into $R^n$ say. Your $f(x_i)$ seems to be $x_i^{'}-Hx_i$ and there would be 8 vectors of this type (for $i=1,2,3,4$). – AnyAD Jun 4 '18 at 15:13

The 2 separate residuals for each data point can easily be dealt with. If there are $n$ data points, return a vector of residuals of length $2n$. For each data point, there are 2 corresponding residuals in the vector of residuals. It all works out correctly, because the overall objective is the sum of squares of all the residuals (2 per data point).