Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I have a series of observations made against time. Given perfect observations, the trend should be reasonably linear. As I expect there to be measurement errors, I wish to exclude them from further calculations.

The time values are not evenly spaced either.

How would I go about identifying which values fall significantly outside of the norm here?

Below is an extract of the table:

Time (hrs)   Observation
21.516       15.071
21.568       11.555
21.614       12.601
21.675       14.194
21.731       11.308
21.787       11.968
21.842       14.383
23.493        3.269
23.537        5.917
23.581        5.982
23.625        5.696
23.669       10.297
23.713        9.599
23.756        4.074
share|improve this question
1  
One approach is to use Least Squares to find a line of best fit, then subtract the observed values from the values approximated by the line of best fit. –  Ben Nov 11 '12 at 15:47

1 Answer 1

up vote 1 down vote accepted

Assuming you are looking for an automated method - This is a big topic, but here are three cheapo methods off the top of my head:

(1) Get a linear least squares fit for the line (in matlab that's three characters of code A\b), look at all the points that have a large distance to this line - they are outliers. However since this line is very skewed towards the outliers, it isn't out of the question that this results in very bad identification of outliers...

(2) Same as above but get a least absolute deviations fit (absolute value vs squared absolute value). That's similar but just slightly less sensitive to outliers.

(3) The best way to avoid outliers is to do a RANSAC fit, as is very common in computer vision for example. Wikipedia RANSAC

share|improve this answer
    
Thanks for that Peter. I've used option 2 and the results are pretty clear to me now. –  Carl Nov 11 '12 at 17:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.