I have a bunch of real world data sets and from manually plotting some of the data in graphs, I've discovered some data sets look pretty much logarithmic and some look linear, or exponential (and some look like a mess :).

I've been reading up on curve fitting / data fitting on wikipedia and if I understand it correctly (which I seriously doubt) I can calculate a curve of best fit using least squares calculations, but I have to determine if I want to have the curve fit a logarithm, linear or exponential (etc) pattern first.

What I would really like to do is to pass a data set into a function (I'm a programmer with poor math skills) and have that return something like "this data set looks more linear than logarithmic" or "this looks exponential".

My question is: is that even possible, without a human looking at a graph and recognizing the pattern ?

My guess is: yes. But before I invest a ton of time in figuring out how to program this, I just want to make sure I'm not barking up the wrong tree and confirm this with you guys if possible.

Sorry if this is a dumb question, but just to be clear, I'm not looking for a how-to answer, just a simple yes or no will do, however if you have suggestions on how to tackle the problem, that would be awesome of course.

  • 1
    $\begingroup$ You can do least squares interpolation using any combination of functions of your choosing. $\endgroup$ – Ryan Budney Apr 8 '11 at 0:03
  • $\begingroup$ There's even it seems a commercial solution for this: uniphiz.com/best-fit.htm though I haven't tried it. The companion program uniphiz.com/findgraph.htm can even recover the data from a plot. $\endgroup$ – lhf Apr 8 '11 at 0:09
  • $\begingroup$ Yes, I can rape the data to fit any line, but how do I know which type of curve 'fits best' $\endgroup$ – Michael Apr 8 '11 at 0:12
  • $\begingroup$ @lhf thanks, but I need to program it myself in php :( $\endgroup$ – Michael Apr 8 '11 at 0:15
  • 1
    $\begingroup$ @Mitch Probably mistyped "shape the data..." $\endgroup$ – Display name Nov 13 '18 at 1:26

You may not realize it but this is a statistics question, for which statisticians have been studying...forever.

So the simple answer to your question 'Can you do it?' is yes.

But of course there's more nuance to that.

Normally statisticians will say 'pick your model first, one of those three (linear, exponential or logarithmic), and then for one of those, I can tell you what the 'best' line is. That is plain old linear regression will give you the best-fit line for a linear model, and for the other two you can transform the data (take a log or exponential of the $y$ value and -then- do linear regression. Part of the process/output of doing linear regression is a value that says how good the match is (the correlation coefficient).

But your question is more along the lines of which 'model' is the best. You might think that you would just compare the three correlation coefficients and pick the best one. I would think that too, except I am not a statistician and something tells me a statistician would have a fit over something so simple (probably also over my suggestion of using the correlation coefficient). So, for a real answer, I think a statistician would be able to answer much better (hmm...isn't there a statistics.stackexchange?). But for the moment, this is a good first approximation to an answer.

Edit: In the meantime, I asked directly at stats.stackexchange (as suggested by Rahul). They confirm my suspicions that a simple answer is yes, but it's not so simple.

  • $\begingroup$ Thank you very much for your answer. Indeed my question is more along the lines of which 'model' is the best. I'm happy with the fact that 'Can you do it?' is yes. I think I'll just have to hire a real statistician to help me implement it. $\endgroup$ – Michael Apr 8 '11 at 1:32
  • $\begingroup$ @Michael, @Mitch: There is indeed a stats.stackexchange.com which may be a good fit for this question. $\endgroup$ – Rahul Apr 8 '11 at 2:18
  • 2
    $\begingroup$ As Mitch says, the fitting to a given model is a well-defined algorithm, so you can do it with each of a set of models. Then if you have a good estimation of the probability of error in each measurement, you can calculate a chi-square and choose the model that has the lowest. The unanswered questions are 1)does your set of models include the true one and 2)is your model of the probability of an error correct? Both are hard questions. $\endgroup$ – Ross Millikan Apr 8 '11 at 3:41
  • $\begingroup$ Thank you all for your insights. $\endgroup$ – Michael Apr 8 '11 at 11:48
  • $\begingroup$ @Rahul: THanks, I took you up on your suggestion. See my link in my edited answer. $\endgroup$ – Mitch Apr 8 '11 at 16:27

You can use a program to plot your data and look at the points. Most plotting programs allow e.g. linear, log-linear and log-log plots (and possibly others). That should give you an idea.

If you know the data, perhaps there is some (approximate?) model of it that hints at a particular functional relationship. In that case I'd set up a least squares fit.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.