looking for a good way to determine if data is linear

I have a big set of data. Each set could be linear or not. I need to find a way to pick out the data set which is linear. Here is what I did.

1) I calculate the slope with the end points

2) from the slope, I can setup a straight line $y(n)=n\times \text{slope} + y_0$, where $y_0$ is the first point of the data.

3) from the data $z(n)$, I can find the difference between each point such that $\Delta(n)=z(n)-y(n)$

4) find the standard deviation of $\Delta(n)$ and accept the data set if the standard deviation is less than 1.

I apply this to all my data sets and it works pretty good so far. But the efficient is pretty low since I have too many data sets (at least 100,000) and for each data set, there are about 100,000,000 samples.

• That's not how you normally find the standard deviation. Read up about linear regression to see the right way. – K. Rmth Feb 2 '14 at 23:33
• Here, do you have the guarantee that the linear set is exactly linear? – Clement C. Feb 2 '14 at 23:34
• the data set is either perfect linear or pretty random – user1285419 Feb 3 '14 at 1:44

The correlation coefficient $r$ is a quick way to check if data is approximately linear or not.
It's equal to $$\frac{1}{n-1} \sum \left( \frac{x - \overline x}{S_x} \right) \left( \frac{y - \overline y}{S_y} \right)$$
• so $x$ is my data set and what is $y$? – user1285419 Feb 3 '14 at 1:45
• Each point of your data set should be in form $(x,y)$ if there are 2 variables – qwr Feb 3 '14 at 1:55
• ok, it makes sense. But what range should r fall into so I can say the data is linear. In my case, x is always linear but y is calculated by a recurrence relation. So to calculate $S_y$ and $\overline{y}$, so the algorithm complexity is about the same as my current one. I am looking for faster way. – user1285419 Feb 3 '14 at 2:04
• Linear or not depends on context, but the closer you are to 1 or -1, the more linear. I want to ask how $x$ can always be linear though, because $x$ is one variable and linear only applies to two variables? – qwr Feb 3 '14 at 2:15
• There is a recurrence relation $v(n+1) = f(a, v(n))$, where $n$ is discrete time which is integer n=0,1,2,3 ..., N. So in my case, above recurrence function might give linear data v's depends on parameter a. In my data, there will only give linear v when for some functional form and some parameters a's. – user1285419 Feb 3 '14 at 2:26