# Exponential least squares biased to small y?

I'm working on fitting an exponential decay curve to a data set. While searching for techniques I found Wolfram's page describing how to easily accomplish it by taking the natural log of the predictor equation. http://mathworld.wolfram.com/LeastSquaresFittingExponential.html

I'm confused by equations (5) and beyond, as there is no justification to the claim that the prediction gives greater weight to small y values. In order to test the claim, I generated a sample data set (took an exponential decay function and added random noise) and compared the predictive model generated by the standard linear regression vs. the one suggested by equations (5) and beyond. I found that the sum of the residuals squared was significantly smaller (6 million vs. 11 million) for the predictive model generated by using the parameters calculated from (9) and (10).

Can someone give me a reason for this bias? Is multiplying by the response (y) just a convenient choice, or can it be shown that it is the optimal correction?

Also, if this is indeed a known bias, why doesn't software designed to generate exponential fits take it into account? I generated the fit using Oracle (REGR_SLOPE, REGR_INTERCEPT) and Excel (Trendline) and both gave me the same fit, which wasn't as good as the one I generated by equations (9) and (10) on Wolfram's site, based on my analysis of the residuals.

• This is just a trick ("to weight the points equally", as they say). What you should do is just linearize the model, get the parameters from linear regression and start a nonlinear regression using these parameters. You cannot compare SSQ for two different objective functions. Sep 24, 2014 at 10:15
• That does get me to a better fit than either of the fits I generated previously. I'm not sure what you mean when you say I can't compare the SSQ. I'm comparing the gross sum of squares calculated by taking the square of the difference between my predicting function and the existing data for every point. Are you saying that this comparison is invalid? Sep 24, 2014 at 21:58
• Could you send me exactly your data points at [email protected] [as a .txt file - one pair of (x,y) per line] and I shall show you things. Cheers :-) Sep 25, 2014 at 3:01

First of all, parameters of unweighted linear regression are very strongly influenced by the largest values of $y$, the dependent variable. When you linearize an exponential model and you have very small values, then the $\log(y)$ can be very large (negative) and strongly influence the results.

I generated $20$ equally spaced $x$'s $(x_i=1,2,3,\cdots,20)$ and generated the corresponding $y$ using $$y=e^{5.678-0.456 x}$$ to which I added a random noise for a maximum relative error of $\pm10$%.

For the first run, I just linearized the system and used the classical linear regression. The resulting parameters are $5.710$ and $-0.460$. For these values was computed $$SSQ=\sum_{i=1}^{20} \Big(y_i^{calc}-y_i^{exp}\Big)^2$$ which is equal to $87.07$.

In a second step, I used the weighting as suggested in the link you give. The resulting parameters are $5.655$ and $-0.443$ and $SSQ=26.64$.

In a third step, starting with the parameters obtained by the first step, I performed a nonlinear regression. The resulting parameters are $5.656$ and $-0.444$ and $SSQ=26.54$.

As you can see, what is proposed leads to results which are quite close to the nonlinear regression.

I must confess that I always perform a nonlinear regression even if the model can be linearized and even if errors are small. Remember that what is measued is $y$ and not $\log(y)$ and when you compare sum of squares they must be consistent.

By the way, you could demonstrate easily that, if the errors are not too large, minimizing the sum of the squares of logarithms is almost identical to minimizing the sum of squares of relative errors. For illustration purposes, I also made that. The resulting parameters are $5.706$ and $-0.460$ and $SSQ=74.74$. Compare these results to those from the first step.

• Your results are very similar to mine. I got only minor improvements beyond Wolfram's suggestion, but it can still be improved upon, so obviously it's not a closed form of the best fit. I do wonder why Excel doesn't iterate to generate a better fit, but maybe they prefer to present a result that can be derived by hand. Oct 4, 2014 at 5:15
• How did you find the correct weights?
– Leo
Feb 17, 2016 at 7:46