12
$\begingroup$

To define outliers, why we cannot use: Lower Limit: Q1-1xIQR Upper Limit: Q3+1xIQR

OR

Lower Limit: Q1-2xIQR Upper Limit: Q3+2xIQR

$\endgroup$
1
  • $\begingroup$ One can surely, change this limit depending upon how much data, he/she is willing to consider as the outlier. These commonly used limits make sure that '0.7%' of data is treated as an outlier; if there's any data point in that region. $\endgroup$ – Dr Nisha Arora Nov 1 '19 at 15:09
12
$\begingroup$

By definition, 50% of all measurements are within $\pm0.5IQR$ of the median. Compare this - heuristically - with a normal distributions where 68% are within $\pm\sigma$, so in that case IQR would be slightly less than $\sigma$. Cutting at $\pm 1.5IQR$ is therefore somewhat comparable to cutting slightly below $\pm3\sigma$, which would declare about 1% of measurements outliers. This matches quite well with the habit of using "$3\sigma$" as a bound in many simple statistical tests. On the other hand, cutting at $\pm1IQR$ would be like cutting near $\pm 2\sigma$, making about 5% outliers - too many; and cutting at $\pm2IQR$ would be like cutting at $\pm4\sigma$, thus turning even many quite extreme measurements into non-outliers. So $\pm 1.5IQR$ is also what Goldilocks would choose.

$\endgroup$
5
  • 1
    $\begingroup$ I don't understand that " IQR would be slightly less than σ". Because one is IQR while another one is Natural distribution, how can you convert one side to another so that you can compare. Also, how did you calculate the 1% and 5%? $\endgroup$ – Fan Oct 10 '14 at 14:41
  • 2
    $\begingroup$ "By definition, 50% of all measurements are within ±0.5IQR of the median." This is true if the median is halfway between the first and third quartiles, but not necessarily otherwise. $\endgroup$ – user856 Jun 21 '16 at 18:27
  • 1
    $\begingroup$ The phrasing $\pm 1.5 IQR$ seems to imply that something is an outlier if it's more than $1.5 IQR$ away from the median, which is not correct. $\endgroup$ – Richard Rast Jul 9 '18 at 13:11
  • $\begingroup$ @RichardRast You are right. $\endgroup$ – Md. Sabbir Ahmed Apr 23 '20 at 16:45
  • $\begingroup$ @hagen-von-eitzen, are you sure about the statement that anything above $\pm 3 \sigma$ comprises about 1% of the observations? Don't you mean about ~0.3%? $\endgroup$ – Sos Sep 14 '20 at 14:32
6
$\begingroup$

The 3rd quartile (Q3) is positioned at .675 SD (std deviation, sigma) for a normal distribution. The IQR (Q3 - Q1) represents 2 x .675 SD = 1.35 SD. The outlier fence is determined by adding Q3 to 1.5 x IQR, i.e., .675 SD + 1.5 x 1.35 SD = 2.7 SD. This level would declare .7% of the measurements to be outliers.

$\endgroup$
0
5
$\begingroup$

We certainly CAN use whatever outlier bound we wish to use, but we will have to justify it eventually. In the not-so-recent past, it was typical to expect distributions to be Gaussian. With that assumption, ±1IQR is too exclusive, resulting in too MANY outliers, ±2IQR is too inclusive, resulting in too FEW outliers. ±1.5IQR is easy to remember, and is a reasonable compromise, under assumptions of Gaussianity.

However, for your distribution and expected outlier fraction, those assumptions may not be appropriate. Additionally, perhaps the definition of outlier is incorrect for your problem, and requires greater detail than just how it behaves within the bounds of a single metric?

$\endgroup$
1
$\begingroup$

As I recall, Prof. Michael Starbird, in one of his lectures in the recorded series, Joy of Thinking: The Beauty and Power of Classical Mathematical Ideas, answers this question. Dr. Starbird reports having attended the very conference presentation in which Tukey introduced this test, and during which Tukey himself was asked this very question. Tukey's answer: two seems like too much and one seems like not enough.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.