2
$\begingroup$

There is a fairly standard technique of removing outliers from a sample by using standard deviation.

Specifically, the technique is - remove from the sample dataset any points that lie 1(or 2, or 3) standard deviations (the usual unbiased stdev) away from the sample's mean. Is it possible with this technique that one ends up removing all points from the dataset. Or, is there a property of sample stdev that prevents this from happening?

$\endgroup$
1
  • $\begingroup$ Since the standard deviation is in a sense a measure of how far from the mean the typical observation is (in a root mean square sense), throwing out observations which are one standard deviation away would be throwing out typical observations rather than outliers. $\endgroup$
    – Henry
    Feb 28, 2022 at 9:34

2 Answers 2

2
$\begingroup$

Duh. Its quite simple to show that there is atleast one datapoint from a sample lying within one stddev from the mean.

Proof -

Assume all datapoints are more than one stddev away from the mean. That is -

$|x_i-\mu| > \sigma$, for all $1 \le i \le n$

Then we have,

$\sum\limits_{i=1}^n(x_i-\mu)^2 > n\sigma^2$

which is in contradiction with the definition of $\sigma$ (sample standard deviation).

$(n-1)\sigma^2 = \sum\limits_{i=1}^n(x_i-\mu)^2$

$\endgroup$
0
$\begingroup$

Interesting question.

For $k>1$, the chevyshev inequality guarantees that we will always have a non-trivial proportion of data points left. If we do this procedure a finite number of times, then some data points will always be left.

We could talk about taking the limit to infinity. At which point, simply show that if there is less than $N=k^2$ points of data, since $\frac{N-1}{N} < 1 - \frac{1}{k^2}$, this implies that we cannot lose any data points in this procedure.

For $k=1$, note that the uniform distribution on 2 points will result in both points lying exactly 1 SD away. Arguably, you'd want to throw this out, and hence we are left with 0 points.

For $k<1$, the uniform distribution on 2 points will result in both points being thrown out. Of course, several other distributions work too, esp those that are very heavy in the tail ends. Note that chevyshev becomes trivial, and hence doesn't apply in this case.

$\endgroup$
3
  • $\begingroup$ Doesn't Chebyshev's inequality deal with populations? My question is about samples. $\endgroup$
    – user869081
    Jun 26, 2013 at 8:32
  • $\begingroup$ The main difference would be that your unbiased estimator of population sd is slightly larger than the actual sample sd. The above applies to the actual sample sd, and you can do the same with it, accounting for the small change. So yes, it's conceivable that if your sample data is very heavy in the tails (e.g. uniform distribution on 2 points), then due to the larger sd used, you start to eliminate all the data. $\endgroup$
    – Calvin Lin
    Jun 26, 2013 at 14:41
  • $\begingroup$ There will always be at least one observation no more than one sample standard deviation away from the sample average, even without using Bessel's correction. If you do use Bessel's correction, the sample standard deviation will become larger so there will always be at least one observation strictly less than one standard deviation away from the sample average (noting that the sample average and standard deviation of the remaining data will change as you remove "outliers") $\endgroup$
    – Henry
    Feb 28, 2022 at 9:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .