# Nonlinear Least Squares Fit to Custom Function in C#

I have a pretty straightforward regression problem I need to solve within the .NET framework in C#. I have a nonlinear transfer function $$Z = A + \frac{1}{\frac{1}{\frac{X}{256}*B+3*C}+\frac{1}{\frac{Y}{1024}*D+2*E}}$$ with two independent variables $$X, Y$$ and five parameters $$A, B, C, D, E$$ for which I need to optimize.

I've been able to successfully model and solve this problem with MATLAB's fit and Python Scipy's curve_fit with very accurate results. However, I have not been able to find a solution available to C#. There are a couple MIT licensed Nuget packages available but they either only support a single independent variable/no custom function (Math.NET) or have poor results (Accord.NET - which may very well be my fault as evidenced here).

I was hoping someone here would have some ideas for alternative solutions or could point me in the right direction in writing my own custom method to solve this. MATLAB used the Trust Region Reflective algorithm and Scipy used either Levenberg-Marquardt or Trust Region Reflective according to their documentation. The data I am working with is

$$\begin{gather} X = \{0, 128, 255, 0, 128, 255, 0, 128, 255\}\\ Y = \{0, 0, 0, 512, 512, 512, 1023, 1023, 1023\}\\ Z = \{89.66623397, 122.9866434, 123.8610312, 197.274736, 4255.419371, 7129.346848, 197.8635428, 4655.314692, 8335.298909\} \end{gather}$$

with a best guess of

$$\begin{gather} A = 20\\ B = 10000\\ C = 50\\ D = 50000\\ E = 60 \end{gather}$$

and I would expect to get

$$\begin{gather} A = 27.85\\ B = 9886.98\\ C = 56.87\\ D = 48581\\ E = 48.47 \end{gather}$$

Any direction or ideas are greatly appreciated.

• This is not what I should call "a pretty straightforward regression problem" – Claude Leibovici Jun 10 '19 at 7:23
• The question is well posed. The OP shows an effort to solve a concret case with given data. I cannot understand why some people voted to close it. This is in contradiction with the rules. So, I vote to reopen. – JJacquelin Jun 16 '19 at 5:43
• @JJacquelin They voted to close it because they think it's not about mathematics. – YuiTo Cheng Jun 16 '19 at 6:18
• @YuiTo Cheng. So, do they think that "Regression", "Least squares" , "Non linear optimization" are not mathematics ? Then why these topics are actually referenced as belonging to the Mathematics StackExchange forum ? – JJacquelin Jun 16 '19 at 6:54
• @Claude Leibovici. Hi Claude! I agree with your comment : "This is not a pretty straightforward regression problem". But "not straightforward" doesn't always mean impossible… Cheers ! – JJacquelin Jun 16 '19 at 11:02

$$Z = A + \frac{1}{\frac{1}{\frac{X}{256}*B+3*C}+\frac{1}{\frac{Y}{1024}*D+2*E}} \tag 1$$ Simplification of writing with $$\begin{cases} a=A \\ b=\frac{B}{256}\\ c=3C \\ d=\frac{D}{1024}\\ e=2E \end{cases}$$ $$Z = a + \frac{1}{\frac{1}{b\,X+c}+\frac{1}{d\,Y+e}}$$

$$(Z-a)(b\,X+c+d\,Y+e) = (b\,X+c)(d\,Y+e)$$

$$bdXY-bXZ-dYZ+b(a+e)X+d(a+c)Y-(e+c)Z+ac+ae+ce=0$$

This is the equation of a quadratic surface : $$a_{110}XY+a_{101}XZ+a_{011}YZ+a_{100}X+a_{010}Y+a_{001}Z=1 \tag 2$$

$$a_{110}=-\frac{bd}{ac+ae+ce}\quad;\quad a_{101}=\frac{b}{ac+ae+ce}\quad;\quad a_{011}=\frac{d}{ac+ae+ce}\quad;\quad a_{100}=-\frac{b(a+e)}{ac+ae+ce}\quad;\quad a_{010}=-\frac{d(a+c)}{ac+ae+ce}\quad;\quad a_{001}=\frac{e+c}{ac+ae+ce}$$

So, this is a problem of regression to fit a quadratic surface. But not a general quadratic surface since the equation doesn't includes all coefficients of the general quadratic surface. http://mathworld.wolfram.com/QuadraticSurface.html

This kind of regression is roughly considered in the paper https://fr.scribd.com/doc/14819165/Regressions-coniques-quadriques-circulaire-spherique page 17. All the cases are not detailed but the method is the same. This is applied in the numerical example below, with the data from the OP.

One can see that the result is quite exactly the result expected by the OP : $$\begin{gather} A = 27.85\\ B = 9886.98\\ C = 56.87\\ D = 48581\\ E = 48.47 \end{gather}$$

DISCUSSION :

This result is unusually excellent for a regression problem. This is certainly because the data doesn't come from experiment but comes from numerical simulation without scatter.

If the data was scattered the result would be not so accurate due to several causes :

• Of course the scatter it-self.

• The number of independent parameters in the equation $$(2)$$ of the particular quadratic surface is six while the number of parameters in the original equation $$(1)$$ is five. It doesn't matter if the scatter is low. But the more the scatter is important the more the optimization differs from the cases $$(1)$$ and $$(2)$$. Also the presence of an extra-parameter eventually makes the calculus less robust.

• The criteria of fitting for the above method is not the least mean square with respect to the data $$Z$$, but is a least mean square with respect to a the right term of Eq.$$(2)$$. Again this doesn't matter if the scatter is low. But if the scatter is not negligible, a criteria of fitting has to be specified and taken into account.

As a conclusion, in case of data coming from real measurements, such above oversimplified method is not sufficient. The above method can produce a rough first fit. The obtained values of the parameters are probably more accurate and more reliable than "guessed" values. They could be used for starting an iterative process of non-linear regression which anyways cannot be avoid to take into account of a particular criteria of fitting.

• I did indeed use unscattered data as a way to compare the results of, as you mentioned, iterative non-linear regression. The data is the ideal case I was using to evaluate various solutions' ability to solve the simple case after which I was planning on introducing real world measurements to assess robustness. I had originally approached this problem algebraically which is doable in the ideal case. However, as you might expect, I quickly ran in to problems with the scattered data. – benavidezb Jun 17 '19 at 21:59
• Any ideas on appropriate iterative non-linear regression techniques? My next approach is to write my own implementation based on a Levenberg-Marquardt paper, if possible. – benavidezb Jun 17 '19 at 22:07
• You can find in the literature a lot of ideas about iterative non-linear regression, from general principle : mathworld.wolfram.com/NonlinearLeastSquaresFitting.htmlhttp://… to many alternatives to the Gauss-Newton method, for example Levenberg-Marquardt en.wikipedia.org/wiki/Levenberg%E2%80%93Marquardt_algorithm . – JJacquelin Jun 18 '19 at 7:01
• The difficulty is not to implement such algorithms. The difficulty is to guess sufficiently correct initial values of the parameters to be optimized. Probably the difficulty that you faced is due to the guessing. Try your own algorithm with better initial guess. I gave you a very simple method in case of the concerned function. Apply it and see. That's all I can do to help you. – JJacquelin Jun 18 '19 at 7:02