16
$\begingroup$

To define outliers, why we cannot use: Lower Limit: Q1-1xIQR Upper Limit: Q3+1xIQR

OR

Lower Limit: Q1-2xIQR Upper Limit: Q3+2xIQR

$\endgroup$
1
  • $\begingroup$ One can surely, change this limit depending upon how much data, he/she is willing to consider as the outlier. These commonly used limits make sure that '0.7%' of data is treated as an outlier; if there's any data point in that region. $\endgroup$ Nov 1, 2019 at 15:09

4 Answers 4

13
$\begingroup$

By definition, 50% of all measurements are within $\pm0.5IQR$ of the median. Compare this - heuristically - with a normal distributions where 68% are within $\pm\sigma$, so in that case IQR would be slightly less than $\sigma$. Cutting at $\pm 1.5IQR$ is therefore somewhat comparable to cutting slightly below $\pm3\sigma$, which would declare about 1% of measurements outliers. This matches quite well with the habit of using "$3\sigma$" as a bound in many simple statistical tests. On the other hand, cutting at $\pm1IQR$ would be like cutting near $\pm 2\sigma$, making about 5% outliers - too many; and cutting at $\pm2IQR$ would be like cutting at $\pm4\sigma$, thus turning even many quite extreme measurements into non-outliers. So $\pm 1.5IQR$ is also what Goldilocks would choose.

$\endgroup$
5
  • 1
    $\begingroup$ I don't understand that " IQR would be slightly less than σ". Because one is IQR while another one is Natural distribution, how can you convert one side to another so that you can compare. Also, how did you calculate the 1% and 5%? $\endgroup$
    – Fan
    Oct 10, 2014 at 14:41
  • 3
    $\begingroup$ "By definition, 50% of all measurements are within ±0.5IQR of the median." This is true if the median is halfway between the first and third quartiles, but not necessarily otherwise. $\endgroup$
    – user856
    Jun 21, 2016 at 18:27
  • 1
    $\begingroup$ The phrasing $\pm 1.5 IQR$ seems to imply that something is an outlier if it's more than $1.5 IQR$ away from the median, which is not correct. $\endgroup$ Jul 9, 2018 at 13:11
  • $\begingroup$ @RichardRast You are right. $\endgroup$ Apr 23, 2020 at 16:45
  • $\begingroup$ @hagen-von-eitzen, are you sure about the statement that anything above $\pm 3 \sigma$ comprises about 1% of the observations? Don't you mean about ~0.3%? $\endgroup$
    – Sos
    Sep 14, 2020 at 14:32
6
$\begingroup$

The 3rd quartile (Q3) is positioned at .675 SD (std deviation, sigma) for a normal distribution. The IQR (Q3 - Q1) represents 2 x .675 SD = 1.35 SD. The outlier fence is determined by adding Q3 to 1.5 x IQR, i.e., .675 SD + 1.5 x 1.35 SD = 2.7 SD. This level would declare .7% of the measurements to be outliers.

$\endgroup$
0
5
$\begingroup$

We certainly CAN use whatever outlier bound we wish to use, but we will have to justify it eventually. In the not-so-recent past, it was typical to expect distributions to be Gaussian. With that assumption, ±1IQR is too exclusive, resulting in too MANY outliers, ±2IQR is too inclusive, resulting in too FEW outliers. ±1.5IQR is easy to remember, and is a reasonable compromise, under assumptions of Gaussianity.

However, for your distribution and expected outlier fraction, those assumptions may not be appropriate. Additionally, perhaps the definition of outlier is incorrect for your problem, and requires greater detail than just how it behaves within the bounds of a single metric?

$\endgroup$
1
  • $\begingroup$ How was it decided what is considered too many and what is considered too few? Or was it just a finger in the air estimate that works well? $\endgroup$ Jan 8 at 17:51
3
$\begingroup$

As I recall, Prof. Michael Starbird, in one of his lectures in the recorded series, Joy of Thinking: The Beauty and Power of Classical Mathematical Ideas, answers this question. Dr. Starbird reports having attended the very conference presentation in which Tukey introduced this test, and during which Tukey himself was asked this very question. Tukey's answer: two seems like too much and one seems like not enough.

$\endgroup$
4
  • $\begingroup$ CORRECTION (This post should REPLACE my post from 2020-03-10): Prof. Murray H. Siegel, in Lecture 20 of The Joy of Mathematics (Part II) said, "I was present when Dr. John Tukey gave a talk, and he was asked... why one point five was chosen [for the IQR multiplier].... what he [Tukey] said was [that] two was too big... and one was too small... and one point five was just right." $\endgroup$
    – Ana Nimbus
    May 27, 2021 at 18:01
  • $\begingroup$ So just a finger in the air estimate that works well enough? $\endgroup$ Jan 8 at 17:52
  • 1
    $\begingroup$ @TimoHuovinen , in my experience, real data are messy and there is a lot of human judgement necessary in order to arrive at actionable conclusions about the data. This has to do with the application of the data and the associated conclusions as much as it has to do with the data itself. In many (almost all?) practical applications, a simple screening process using an estimate of what the "correct" IQR multiplier should be is workable. As a reminder, the screening process that is the Tukey Test is just that, a screening process. $\endgroup$
    – Ana Nimbus
    Jan 10 at 20:01
  • $\begingroup$ Thank you, I'm just getting into statistics and it's very useful to understand how things operate and the origin of things. $\endgroup$ Jan 11 at 8:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.