Exponential least squares biased to small y?

Question

I'm working on fitting an exponential decay curve to a data set. While searching for techniques I found Wolfram's page describing how to easily accomplish it by taking the natural log of the predictor equation. http://mathworld.wolfram.com/LeastSquaresFittingExponential.html

I'm confused by equations (5) and beyond, as there is no justification to the claim that the prediction gives greater weight to small y values. In order to test the claim, I generated a sample data set (took an exponential decay function and added random noise) and compared the predictive model generated by the standard linear regression vs. the one suggested by equations (5) and beyond. I found that the sum of the residuals squared was significantly smaller (6 million vs. 11 million) for the predictive model generated by using the parameters calculated from (9) and (10).

Can someone give me a reason for this bias? Is multiplying by the response (y) just a convenient choice, or can it be shown that it is the optimal correction?

Also, if this is indeed a known bias, why doesn't software designed to generate exponential fits take it into account? I generated the fit using Oracle (REGR_SLOPE, REGR_INTERCEPT) and Excel (Trendline) and both gave me the same fit, which wasn't as good as the one I generated by equations (9) and (10) on Wolfram's site, based on my analysis of the residuals.

Answer 1

First of all, parameters of unweighted linear regression are very strongly influenced by the largest values of $y$, the dependent variable. When you linearize an exponential model and you have very small values, then the $\log(y)$ can be very large (negative) and strongly influence the results.

I made an example fo your problem.

I generated $20$ equally spaced $x$'s $(x_i=1,2,3,\cdots,20)$ and generated the corresponding $y$ using $$y=e^{5.678-0.456 x}$$ to which I added a random noise for a maximum relative error of $\pm10$%.

For the first run, I just linearized the system and used the classical linear regression. The resulting parameters are $5.710$ and $-0.460$. For these values was computed $$SSQ=\sum_{i=1}^{20} \Big(y_i^{calc}-y_i^{exp}\Big)^2$$ which is equal to $87.07$.

In a second step, I used the weighting as suggested in the link you give. The resulting parameters are $5.655$ and $-0.443$ and $SSQ=26.64$.

In a third step, starting with the parameters obtained by the first step, I performed a nonlinear regression. The resulting parameters are $5.656$ and $-0.444$ and $SSQ=26.54$.

As you can see, what is proposed leads to results which are quite close to the nonlinear regression.

I must confess that I always perform a nonlinear regression even if the model can be linearized and even if errors are small. Remember that what is measued is $y$ and not $\log(y)$ and when you compare sum of squares they must be consistent.

By the way, you could demonstrate easily that, if the errors are not too large, minimizing the sum of the squares of logarithms is almost identical to minimizing the sum of squares of relative errors. For illustration purposes, I also made that. The resulting parameters are $5.706$ and $-0.460$ and $SSQ=74.74$. Compare these results to those from the first step.