I have a big set of data. Each set could be linear or not. I need to find a way to pick out the data set which is linear. Here is what I did.

1) I calculate the slope with the end points

2) from the slope, I can setup a straight line $y(n)=n\times \text{slope} + y_0$, where $y_0$ is the first point of the data.

3) from the data $z(n)$, I can find the difference between each point such that $\Delta(n)=z(n)-y(n)$

4) find the standard deviation of $\Delta(n)$ and accept the data set if the standard deviation is less than 1.

I apply this to all my data sets and it works pretty good so far. But the efficient is pretty low since I have too many data sets (at least 100,000) and for each data set, there are about 100,000,000 samples.

  • $\begingroup$ That's not how you normally find the standard deviation. Read up about linear regression to see the right way. $\endgroup$ – K. Rmth Feb 2 '14 at 23:33
  • $\begingroup$ Here, do you have the guarantee that the linear set is exactly linear? $\endgroup$ – Clement C. Feb 2 '14 at 23:34
  • $\begingroup$ the data set is either perfect linear or pretty random $\endgroup$ – user1285419 Feb 3 '14 at 1:44

The correlation coefficient $r$ is a quick way to check if data is approximately linear or not.

It's equal to $$\frac{1}{n-1} \sum \left( \frac{x - \overline x}{S_x} \right) \left( \frac{y - \overline y}{S_y} \right) $$

Many statistics programs have this function built in.

  • $\begingroup$ so $x$ is my data set and what is $y$? $\endgroup$ – user1285419 Feb 3 '14 at 1:45
  • $\begingroup$ Each point of your data set should be in form $(x,y)$ if there are 2 variables $\endgroup$ – qwr Feb 3 '14 at 1:55
  • $\begingroup$ ok, it makes sense. But what range should r fall into so I can say the data is linear. In my case, x is always linear but y is calculated by a recurrence relation. So to calculate $S_y$ and $\overline{y}$, so the algorithm complexity is about the same as my current one. I am looking for faster way. $\endgroup$ – user1285419 Feb 3 '14 at 2:04
  • $\begingroup$ Linear or not depends on context, but the closer you are to 1 or -1, the more linear. I want to ask how $x$ can always be linear though, because $x$ is one variable and linear only applies to two variables? $\endgroup$ – qwr Feb 3 '14 at 2:15
  • $\begingroup$ There is a recurrence relation $v(n+1) = f(a, v(n))$, where $n$ is discrete time which is integer n=0,1,2,3 ..., N. So in my case, above recurrence function might give linear data v's depends on parameter a. In my data, there will only give linear v when for some functional form and some parameters a's. $\endgroup$ – user1285419 Feb 3 '14 at 2:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.