Chauvenet's Criterion - All my data points are outliers?

I have runtimes for requests on a webserver. Sometimes events occur that cause the runtimes to skyrocket (we've all seen the occasionaly slow web page before). Sometimes, they plummet, due to terminated connections and other events. I am trying to come up with a consistent method to throw away spurious events so that I can evaluate performance more consistently.

I am trying Chauvenet's Criterion, and I am finding that, in some cases, it claims that all of my data points are outliers. How can this be? Take the following numbers for instance:

[30.0, 38.0, 40.0, 43.0, 45.0, 48.0, 48.0, 51.0, 60.0, 62.0, 69.0, 74.0, 78.0, 80.0, 83.0, 84.0, 86.0, 86.0, 86.0, 87.0, 92.0, 101.0, 103.0, 108.0, 108.0, 109.0, 113.0, 113.0, 114.0, 119.0, 123.0, 127.0, 128.0, 130.0, 131.0, 133.0, 138.0, 139.0, 140.0, 148.0, 149.0, 150.0, 150.0, 164.0, 171.0, 177.0, 180.0, 182.0, 191.0, 200.0, 204.0, 205.0, 208.0, 210.0, 227.0, 238.0, 244.0, 249.0, 279.0, 360.0, 378.0, 394.0, 403.0, 489.0, 532.0, 533.0, 545.0, 569.0, 589.0, 761.0, 794.0, 1014.0, 1393.0]


73 values. A mean of 222.29, and a standard deviation of 236.87. Chauvenet's criterion for the value 227 would have me calculate the probability according to a normal distribution (0.001684 if my math is correct). That number times 73 is .123, less than .5 and thus an outlier. What am I doing wrong here? Is there a better approach that I should be taking?

-
How do you get 0.001684? The probability that a sample from a normal distribution is at least $\frac{5}{236}$ standard deviations away from the mean ought to be very close to 1. – Henning Makholm Sep 17 '12 at 16:53
It would not surprise me in the least to find out that I did that horribly wrong. I pulled it from wikipedia (en.wikipedia.org/wiki/Normal_distribution#Definition). The Chauvenet's definition I read said to use that (as opposed to the probability that a data point occurs some distance from the mean). – dave mankoff Sep 17 '12 at 16:58

Your data is not close to being normally distributed even if you exclude outliers so the whole procedure is rather dubious.

That being said, if you follow the procedure set out in Wikipedia, then you would reject the two most extreme points.

For example $1014$ is $(1014-222.29)/238.51 \approx 3.34$ standard deviations above the mean.

The probability of being $3.34$ or more standard deviations above or below the mean on a normal distribution is about $2(1-\Phi(|3.34|)) \approx 0.001$ and multiplying this by the number of data points $73$ give a figure less than $0.5$, which is apparently the criterion.

The data point $1393$ is even more extreme, while the datapoint $794$ gives a result above $0.5$.

-
+1 Henry very Good answer! – Michael Chernick Sep 17 '12 at 19:17
Thanks! This solves my problem, but please what is Φ ? – Lisa Jan 29 '15 at 10:20

Methods such as Chauvenet's and Peirce's are 19th century attempts at formal testing for outliers from a normal distirbution. They are not sound approaches because they do not consider the distribution of the largest and smallest values from n iid normally distributed variables. Test's such as Grubbs' and Dixon's do take proper account of the distribution of the extreme order statistics from a normal distribution and should be used over Chauvenet or Peirce. As the wikipedia articles mention outlier detection is different from outlier rejection. Rejecting a data point merely on the basis of an outlier test is controversial and many statisticians including me don't agree with the idea of rejecting outliers based solely on these types of tests.

-
That's a fair point. In my case, however, I am trying to build an automated tool produce some quick summary stats. Some of the giant outliers I see are just noise for such a report. I am definitely building in the ability to choose whether outliers are removed, however. I will take another look at Grubb's as well as Dixon's. – dave mankoff Sep 17 '12 at 20:03
Dixon's ratio test is the easiest test to use. Also as I showed in my paper in American Statistician 1982 it is robust in small samples to departures from normality. This means the test maintains its significance level when the population distributions differ soemwhat from the normal. Grubbs' test is optimal for normal data and so is sensitive to departures from normality. – Michael Chernick Sep 17 '12 at 20:14