# Constrained parameters in non linear curve fitting

I have to fit some data points to the non linear curve defined by:

$$y(x) = a + \frac{b}{(x - c)}$$

with constraints $$a,b,c \geq0$$

So far I've tried gradient descent with penalty methods but I'm not getting correct results. At any point $$(x_0,y_0)$$, my loss function looks like:

$$(y_0 - y(x_0))^2 -r (\log(a) + \log(b) + \log(c))$$

What's the best way to model this?

I was going through Constrained parameters in least square curve fitting, which is similar to my problem. How can I use this approach with the curve I defined?

• You said that you are not getting correct results. In order to understand what is the problem, would you edit an example of data to which the trouble arrises. – JJacquelin May 30 at 7:12
• @JJacquelin Example data would be X = [0.000000000050, 0.000000000075, 0.000000000125, 0.000000000175, 0.000000000275, 0.000000000475, 0.000000001100] Y = [0.791999995708, 0.703999996185, 0.659999996424, 0.615999996662, 0.571999996901, 0.527999997139, 0.483999997377]. – Caty May 31 at 12:11
• Could you provide some information about the background of the problem, and especially why the parameters have to have these restrictions? Does the constraint arise from mathematical considerations that are robust, or is it arbitrarily imposed to fit some physical model? Bayesian modelling should be used if your function is difficult to evaluate. – DinosaurEgg Jun 3 at 23:49

What I'm going to propose is definitely not common but it will definitely help with data that is not very close to the singularity $$c$$, since it penalizes them more than the conventional approach to the loss function and therefore makes them more visible to the minimization procedure. As an example, if you want to fit a number of points on the tail of the curve, then finding the value $$c$$ is certainly unstable, since all the data points you have have a similar loss function and thus the parameters a,b,c will have similar loss functions for a wide range of values, making it hard to distinguish what the right choice is, especially if the problem is non-linear and has several minima.

Consider the loss function

$$L=\sum_i(y_i(x_i-c)-a-b(x_i-c))^2$$

If it is zero, then we recover that the (x,y)'s are on the same curve you wish to use. It is a fortunate coincidence that the minimization problem of this loss function has a unique solution!

$$\frac{\partial L}{\partial a}=\sum_i (y_i(x_i-c)-a-b(x_i-c))=0\\ \frac{\partial L}{\partial b}=\sum_i (y_i(x_i-c)-a-b(x_i-c))(-x_i+c)=0\\ \frac{\partial L}{\partial c}=\sum_i (y_i(x_i-c)-a-b(x_i-c))(-y_i+b)=0$$

We can rewrite these equations in the form (note that $$\overline{WXY}=\frac{\sum_i W_iX_iY_i}{N}$$):

$$a+b\bar{x}+c\bar{y}-bc-\overline{xy}=0\\ a\bar{x}+b\overline{x^2}+c\overline{xy}-bc\bar{x}-\overline{yx^2}=0\\ a\bar{y}+b\overline{xy}+c\bar{y^2}-bc\bar{y}-\overline{y^2x}=0$$

After a miraculous cancellation of the non-linear term $$bc$$ we obtain the unique solution

$$b=\frac{AZ-BY}{XZ-Y^2}\\ c=\frac{BX-AY}{XZ-Y^2}\\ a=-b\bar{x}-c\bar{y}+bc+\overline{xy}$$

where I defined:

$$A=\overline{x^2y}-\overline{xy}~\overline{x}~~,~~B=\overline{y^2x}-\overline{xy}~\overline{y} \\ X=\overline{x^2}-\bar{x}^2~~, ~~Z=\overline{y^2}-\bar{y}^2~~, ~~ Y=\overline{xy}-\bar{x}\bar{y}$$

With this now you have a tool to minimize with analytical control. Hope it works!

• Thanks @DinosaurEgg, but unfortunately it didn't work for me. I got one of the parameter negative and it led to high mean squared error when I tested on the same data I used to fit the curve. – Caty May 31 at 12:15
• What that means is that this is a bad choice of function to do least square regression with. Is there any reason for you to absolutely need c>0? – DinosaurEgg May 31 at 18:49
• This is the demand of problem. If not least square regression could you please guide on other better ways to model this? – Caty Jun 2 at 17:43
• Well if that's your data and you can't change them, maybe try to understand why you're not getting what you think you should be? Maybe get more data? There's about 3 people telling you here that the fit favors negative values because those truly minimize quadratic loss functions. So maybe you should be content with a bad fit that has c=0 set by hand or do something else about it. Not every problem can be fixed by smart regression. – DinosaurEgg Jun 3 at 4:33

Preliminary inspection with the data :

x = [0.000000000050, 0.000000000075, 0.000000000125, 0.000000000175, 0.000000000275, 0.000000000475, 0.000000001100]

y = [0.791999995708, 0.703999996185, 0.659999996424, 0.615999996662, 0.571999996901, 0.527999997139, 0.483999997377]

First I suspected a computation problem due to the low magnitude of the x values.

In order to eliminate any trouble of this kind, the change of variable $$\quad X=10^{10}x\quad$$ transforms the data into :

X = [0.50, 0.75, 1.25, 1.75, 2.75, 4.75, 11.00]

$$y=a+\frac{b}{x-c}\quad$$ is transformed into $$\quad y=a+\frac{B}{X-C}\quad \begin{cases} B=10^{10}b\\ C=10^{10}c \end{cases}$$

A least mean square regression gives : $$\begin{cases} a= 0.45651934\\ B= 0.37824659\quad ;\quad b=0.000000000037824659\\ C=-0.66325\quad ;\quad c=-0.000000000066325 \end{cases}$$

So, the parameter $$C$$ is negative and this is not due to numerical computation with small numbers.

Thus the first hypothesis is eliminated.

By comparison, if we proceed with linear regression with respect to the parameters $$a$$ and $$b$$ for given values of $$C$$ we observe how the fitting depends on $$C$$ :

The deviation is continuously increasing when $$C$$ is increased in order to tend to positive value.

As a consequence if the regression is subjected to a constraint $$\quad c\geq 0\quad$$ one have to accept a very bad fitting.

It is normal that your results from gradient descent with penalty methods are not as good as expected. This is the unavoidable consequence of the constraint in the specific case of the function $$y=a+\frac{b}{x-c}$$.

One can expect a better fitting with constraint in choosing another function, may be with more parameters and/or better convenient as model.

• Thanks a lot @JJacquelin for your detailed explanation. How did you decide on the initial values? As I find my model highly sensitive to the initial values. Also what other models could be convenient? Can Bayesian optimization help? – Caty Jun 2 at 17:41
• A very simple method which doesn't need initial values consists in linear regression for this equation : $$ax-cy+d=xy$$ This leads to $a\simeq 0.447417\:;\: c=-0.8273594\:10^{-10}\:;\: b=d+ac\simeq 0.4380024\:10^{-10}$.This can be used as initial values for non-linear regression. Even without non-linear regression the result is satisfying : RMSE=$0.0098$ instead $0.0092$ with non-linear regression. But with the condition $c\geq 0$ as shown in my main answer we get $c=0\:;\: a\simeq -0.028852 \:;\: b\simeq 1.344938\:e^{-10}\:$ and the fitting cannot be good anyways. – JJacquelin Jun 2 at 19:41
• Search for more convenient models is a different problem without general and sure method. Try other functions, probably with more parameters than three. Often physical consideration of the phenomena which one want to model gives clues about the form of equation to choose. – JJacquelin Jun 2 at 19:52