I have some data shown in the image. Each symbol corresponds to a different dataset. As it can be seen the data initially behaves like a line and after some points it follows other behaviour. enter image description here

The idea is to fit a line that passes through the origin and that the line "is good enough" approximating the behaviour of the data where it is linear.

What I tried is to do the following: I pick a minimum of data (e.g. $m_i$) for each data set $D_i = \{d_{ij}=(x_j,y_j)_i, j\in\{1,...,N_i\}\}$, where $N_i = |D_i|$ is the number of data of each set, and $x_1=0, x_p < x_q$ if $p < q$ . Now, for $D_i$ we consider subsets $S_{ik}=\{d_{ij}=(x_j,y_j)_i, j\in\{1,...,k_i\}\}$, where $k_i \in \{m_i,...,N_i\}$. Each of these subsets has associated with it a correlation coefficient $p_{ik}$. My idea was to pick the maximum of these coefficients and to fit a line $y=mx$ through the data in $S_{ik}$ associated with $\max_k p_{ik}$.

The problem with this is that the correlation coefficient tells how the $y$ data behaves with respect to the $x$ data, i.e. how good a line would fit the data. This line intercepts the $y$ axis at a point that in general is not $0$.

My next try was to instead fit a line $y=mx$ for each $S_{ik}$ and then compute the mean squared error MSE. Then pick the subset that is associated with the minimum value of MSE. This also fails because of course the minimum value is the one associated with the subset with less number of data (MSE penalizes the variation very hard, I look for something "softer").

Is there a way of knowing where the data behaves like a line passing through the origin? I was thinking of taking a small subset of data S_{ik}, fitting a line $y_i=mx$, computing the MSQ. Then adding one data point to the subset and computing its error, then compare it with the MSQ and decide whether to continue the process of adding data, or stopping.

Any help is very much appreciated.

  • $\begingroup$ This will be inherently subjective. Compare to the fact that there is no absolute way of determining what counts as an "outlier" observation. Considering that you're likewise looking to ignore some of the data points, you will need to set the criteria for "minimum accepted correlation" or the like. My instinct here would be to not use linear regression; it looks more like parabolic (quadratic) regression to me, which might more cleanly resolve your problem. $\endgroup$ – Daniel R. Collins Jan 19 '16 at 4:45
  • $\begingroup$ I agree that fitting a straight line to these points looks like the wrong thing to do. But if you really must do this, you will have to tell us where the data comes from and what it will be used for. Otherwise we have no basis for any advice we might offer. $\endgroup$ – TonyK Jan 19 '16 at 11:55
  • $\begingroup$ @DanielR.Collins I'm interested in the linear part of the data because it follows some model. I want to conclude that the model is not useful for some points. But I want to get a way of determining the "best" part of the data that could be fit by a line. I'm aware that this is difficult (maybe there is a way). $\endgroup$ – Vladimir Vargas Jan 20 '16 at 0:17
  • $\begingroup$ @TonyK The data comes from examining the position of the free end of a cantilever subjected to a load at the free end, the load is the x axis and the vertical displacement is the y axis. Physically there is a domain in which the molecular arrangement of the cantilever's material produce a restoring force that makes the data behave linearly, following Hooke's law. After some load, we get that the molecular arrangement is damaged, causing the material to behave plastically (not linearly). $\endgroup$ – Vladimir Vargas Jan 20 '16 at 0:20
  • 1
    $\begingroup$ Thanks for that info. The top set of points, in particular, doesn't look linear over any range $-$ do you have a physical explanation for this? $\endgroup$ – TonyK Jan 20 '16 at 0:22

I worked this out a number of years ago.

If you go the the process of minimizing $\sum (y_i-ax_i)^2$ as a function of $a$, you get $a =\frac{\sum x_iy_i}{\sum x_i^2} $.

Here's how the math works out:

Let $D =\sum_{i=1}^n (y_i-ax_i)^2 $. Then

$\begin{array}\\ \frac{\partial D}{\partial a} &=\sum_{i=1}^n \frac{\partial (y_i-ax_i)^2}{\partial a}\\ &=\sum_{i=1}^n -2x_i(y_i-ax_i)\\ &=-2(\sum_{i=1}^nx_iy_i-a\sum_{i=1}^nx^2_i)\\ &=0 \qquad\text{when }a = \dfrac{\sum_{i=1}^nx_iy_i}{\sum_{i=1}^nx^2_i}\\ \end{array} $

For this value of $a$,

$\begin{array}\\ D &=\sum_{i=1}^n \left(y_i-ax_i\right)^2\\ &=\sum_{i=1}^n \left(y^2_i-2ax_iy_i+a^2x^2_i\right)\\ &=\sum_{i=1}^n y^2_i-2a\sum_{i=1}^nx_iy_i+a^2\sum_{i=1}^nx^2_i\\ &=\sum_{i=1}^n y^2_i-2\dfrac{\sum_{i=1}^nx_iy_i}{\sum_{i=1}^nx^2_i}\sum_{i=1}^nx_iy_i+\left(\dfrac{\sum_{i=1}^nx_iy_i}{\sum_{i=1}^nx^2_i}\right)^2\sum_{i=1}^nx^2_i\\ &=\sum_{i=1}^n y^2_i-\dfrac{\left(\sum_{i=1}^nx_iy_i\right)^2}{\sum_{i=1}^nx^2_i}\\ \end{array} $

Note that both the value of $a$ and the value of $D$ can be easily updated when a data point is either added or removed. That way, you can examine how $D$ changes (though you probably want to compute $\frac1{n}D$) and stop when it gets too large.

  • $\begingroup$ But I'm interested in also knowing where do I "cut" the data to get just the part of it that behaves like a line. $\endgroup$ – Vladimir Vargas Jan 19 '16 at 1:49
  • $\begingroup$ Look at the mean error $\frac1{n}\sum_{i=1}^n (y_i-ax_i)^2$ for each value of $a$ as successive points are added, and stop when that gets too large. $\endgroup$ – marty cohen Jan 19 '16 at 2:54
  • $\begingroup$ Also, see my new, improved answer. $\endgroup$ – marty cohen Jan 19 '16 at 3:15
  • $\begingroup$ marty, I don't see the difference between this approach and the mean squared error. $D$ is just the mean squared error, right?. Thanks for your help though. $\endgroup$ – Vladimir Vargas Jan 19 '16 at 14:57
  • $\begingroup$ Yes it is. But the MSE is easily updatable, so you can monitor it and stop when it gets too large. $\endgroup$ – marty cohen Jan 19 '16 at 17:01

Are you required to use a straight line? It looks to me like a quadratic would fit better and require only a little more trouble.

  • $\begingroup$ Hi, your answer should be a comment. It is some physical data and I'm interested in the linear part. $\endgroup$ – Vladimir Vargas Jan 19 '16 at 14:54
  • $\begingroup$ @VladimirVargas: The trouble is, there doesn't seem to be a linear part. $\endgroup$ – TonyK Jan 19 '16 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.