1
$\begingroup$

Hello StackExchange community!

I am currently a 12th grader who has to research statistics on a post-high school level.

The topic that I was assigned is least-squares regression. So far I have read about and understood the concept behind linear least-squares regression. I then needed to gather data for regression analysis and now that I have gathered sufficient data, I know that the data does not follow a linear model. I, however, am really struggling to understand any other forms of curve fitting apart from data that follows a linear relationship, so here is the situation:

I have a set of points in the form of (x,y) that l can either fit by a hyperbola of the form $y=k/x$, a logarithmic function of the form $y=a*ln(cx+d)+g,$ or a square root function of the form $y=a(\sqrt {bx+c})+d$. All that I have to do using the least-squares method.

Could someone please explain to me (step-by-step and in easy terms) how I would do that? An example would be greatly appreciated! I already read something about linearization and then using linear regression, but frankly speaking I don't quite understand that either, so it'd be great if you could explain that too. You would help out a high-schooler in great despair haha.

Thank you very much in advance!

(P.S I apologize for any semantic or grammatic mistakes. English is not my native language.)

$\endgroup$
0

4 Answers 4

1
$\begingroup$

Linearization is a blanket term which refers to modifying the independent variable so it relates linearly to the dependent variable. The idea is to linearize the data via the three potential models, and then try least squared regression on all of them.

For example, to linearize the hyperbola, introduce a new independent variable $z=\frac1x$. Why is this helpful? Well, note that in the hyperbola model, $y=kz$. So, you can do LS regression on $y$ and $z$ to fit the hyperbolic model.

Likewise, we can linearize $x$ into $z$ to fit the other models.

To linearize the logarithm, let $z=\ln(cx)$, which makes the logarithmic model $y=a\ln(d)\cdot z+g$.

To linearize the square root, let $z=\sqrt{bx+c}$, which makes the square root model $y=az+d$.

$\endgroup$
1
$\begingroup$

Could someone explain to you (step-by-step and in easy terms) how do that?

Whithout wanting to distress you, I would say that is not so simple to be able to explain this wide subject "step-by-step and in easy terms". One have to make effort to learn the basics.

http://mathworld.wolfram.com/NonlinearLeastSquaresFitting.html

https://en.wikipedia.org/wiki/Non-linear_least_squares

In practice it is important to understand that the fitting cannot be good if the shape of the function to fit is far from the shape of the data. It is of interest to first observe a drawing of the points. For example with your data :

enter image description here

We observe that the shape is not far to linear. So the linear regression is the first to try : $y(x)\simeq ax+b$

enter image description here

Trying to fit the function $y(x)\simeq \frac{k}{x}$ is wasting time because obviously the function is not convenient ($k>0$ for positive $y$ thus decreasing function while increasing is required).

Instead of $y(x)\simeq \frac{k}{x}$ one can try a more general form of hyperbolic equation : $$y(x)\simeq\frac{ax+b}{cx+d}\quad\text{with}\quad d=1\quad\text{for non-redondency.}$$ This example allows to show a possible linearisation (among others):

$(cx+1)y\simeq ax+b$ $$a\frac{x}{y}+b\frac{1}{y}+c(-x)\simeq 1$$ A linear regression can be carried out for the coefficients $a,b,c$.

"Linear" means linear with respect of $a,b,c$, of course not wrt the variable terms $\frac{x}{y}\,,\,\frac{1}{y}\,,\,(-x)$.

enter image description here

Note : This is not best fit of $y(x)=\frac{ax+b}{cx+1}$ in the sens of least mean square because it is the best fit for $a\frac{x}{y}+b\frac{1}{y}+c(-x)= 1$ which is not exactly the same criteria of fitting. If we want the least mean squate in strict sens one have to proceed with non-linear regression. This is an iterative process requiring a good initial guess of the parameters. The above values of $a,b,c$ can be taken as very good initial values.

It should be much too long to go into the non-linear regression which suppose to built an algorithm or to use a commercial software.

Case of the function $y(x)=a\sqrt{bx+c}+d$ . It can be lineralized :

$(y-d)^2=a^2bx+a^2c$

$y^2-2d\,y-a^2bx=a^2c-d^2$ $$Ay^2+By+Cx=1$$ $A=\frac{1}{a^2c-d^2}\quad;\quad B=\frac{-2d}{a^2c-d^2}\quad;\quad C=\frac{-a^2b}{a^2c-d^2}$

You can do a linear regression for $A,B,C$. This gives a first result for $a,b,c,d$ in fixing one of them since there is a redondency. If necessary one can improve thanks to a non-linear regression starting from the first values obtained.

I don't continue on this case because I suspect that the function $y(x)=a\sqrt{bx+c}+d$ is not convenient for a good fitting to the given data.

Case of the function $y(x)=a\ln(cx+d)+g$.

Linearisation seems possible but would involve the transformation into an integral equation. Some examples are given in https://fr.scribd.com/doc/14674814/Regressions-et-equations-integrales . This is a non-conventional method. More likely go directly to non-linear regression.

$\endgroup$
4
  • $\begingroup$ Could you please point me to a derivation of the matrices you used? $\endgroup$
    – Christian
    May 21, 2022 at 18:27
  • $\begingroup$ @Christian. Sorry I cannot understand what exactly is your request. What is your problem ? $\endgroup$
    – JJacquelin
    May 22, 2022 at 6:34
  • $\begingroup$ For example I could not find any resources on why for the model $y = ax+b$ $\begin{pmatrix}a \\ b\end{pmatrix} = \begin{pmatrix}\Sigma x^2 & \Sigma x \\ \Sigma x & n\end{pmatrix}^{-1} \begin{pmatrix}\Sigma xy \\ \Sigma x\end{pmatrix}$ $\endgroup$
    – Christian
    May 22, 2022 at 8:02
  • $\begingroup$ @Christian. For the linear model $y=a+bx$ one can find the matrix equation (and proof) in any course about least squares regression. For example eq.(10) in mathworld.wolfram.com/LeastSquaresFitting.html . $\endgroup$
    – JJacquelin
    May 22, 2022 at 8:23
1
$\begingroup$

I post a second answer because the data provided by the OP changed meamwhile.

The scatter is much larger than before which changes a lot the scope.

For the reccord the linear regression leads to :

enter image description here

HYPERBOLIC REGRESSION :

Claude Leibovici already gave a very good answer in the case of an hyperbolic regression. The next graph is drawn from his results :

enter image description here

The scatter is so large that the non-linear regression doesnt improve the fitting. The Root Mean Square Errors are very close to be the same. The graphs (blue curves) are undistinguishable. So, in this case there is no need for a final non-linear regression. The preliminary approximate as Claude Leibovici did is sufficient.

Moreover by comparing to the above linear regression, one see that the RMSE is almost not improved when going from linear regression to hyperbolic regression. So, in this case of large scatter it is of no interest to fit an hyperbolic function instead of a simple linear function.

NOTE : From inspection, one point: $(0.9,3.4)$ appears as an outlier. Eliminating it changes the conclusion.

enter image description here

An even simpler hyperbolic function could be used with almost the same fitting. $$y(x)=\frac{a}{x}+c$$

enter image description here

$\endgroup$
1
  • $\begingroup$ Good answers (as usual) ! Cheers :-) $\endgroup$ Dec 22, 2019 at 5:03
1
$\begingroup$

Being very trivial, I shall say that, if the model is nonlinear with respect to even a single parameter, you will need nonlinear regression or optimization. The problem is that, most of the time, this requires good or, at least, consistent, estimates of the parameters.

Linearization helps in some case. For example, $y=a e^{bx}$ is easy to linearize but $y=a e^{bx}+c$ is not.

In any manner, you need by the end to use nonlinear regression or optimization since what is measured is $y$ and not any of its possible transforms.

If you have one case, let us try together for a small number of data points you could add to your post (even synthetic data with noise).

Edit

After comments, let us consider the following data set $$\left( \begin{array}{cc} x & y \\ 1.40 & 0.33 \\ 2.18 & 2.33 \\ 1.74 & 1.27 \\ 3.64 & 2.43 \\ 3.00 & 3.00 \\ 2.66 & 1.65 \\ 5.34 & 3.65 \\ 6.14 & 3.13 \\ 7.00 & 4.00 \\ 7.80 & 3.99 \\ 8.62 & 3.37 \\ 9.16 & 3.91 \\ 0.90 & 3.40 \end{array} \right)$$ to be fitted according to the model $$y=\frac{k}{b+x}+c$$ In a preliminary step, I should fix $b$ at an arbitrary value and define $t_i=\frac{1}{b+x_i}$. This makes the model to be $y=k t+c$ which is a linear regression. For this value of $b$, compute the sum of squares of the residuals $(SSQ)$.

Running for different values of $b$, you should notice that it is a very flat function of and this is not good sign (already revealed by a scatter plot of the data). Anyway, the minimum value appears around $b=28$ and , at this point, we have $k= -305.56$ and $c=12.25$.

Now, we have all the elements to run the nonlinear regression which will give $b=28.09$, $k= -307.28$ and $c=12.28$ corresponding to $R^2= 0.936$ which is not very good.

$\endgroup$
7
  • $\begingroup$ Hello Mr Leibovici, thanks for your answer! $\endgroup$ Dec 20, 2019 at 7:58
  • $\begingroup$ Here is one set of datapoints: $A(0.1666, 17.2); B(0.5, 23.4); C(1, 30.8); D(1.5, 42,9); E(2, 46.7), F(2.5, 55.8); G(3, 61.5); H(3.5, 63.1), I(4, 59.7)$ $\endgroup$ Dec 20, 2019 at 8:08
  • $\begingroup$ Just to not confuse you: The letters are just names of the datapoints. They have no other meaning attached. How would you then do nonlinear regression (and fit the points to one of the three above models)? $\endgroup$ Dec 20, 2019 at 8:13
  • $\begingroup$ @HermionNuru. I do not think that these data could be properly fitted by any of the models you mention in the post. It could be better you generate some data points according to any of the models using your own parameters (keep them secret), add some noise (for example, rounds the values as you wish). Tell me the model and give me the points. Cheers :-) $\endgroup$ Dec 20, 2019 at 8:45
  • $\begingroup$ Dear Mr Leibovici, please excuse my typo. Als, thanks for your response again! I have the points below: $A(1.4,0.33) B(2.18,2.33) C(1.74,1.27) D(3.64,2.43) E(3,3) F(2.66,1.65) H(5.34,3.65) I(6.14,3.13) J(7,4) K(7.8,3.99) L(8.62,3.37) M(9.16,3.91) N(0.9,34)$. $\endgroup$ Dec 20, 2019 at 13:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .