0
$\begingroup$

I have a list of measured numbers (e. g. lengths of products). Of these I can easily compute the mean and the standard deviation.

Now, when a new measured number arrives, I'd like to tell the probability that this number is of this list or that this number is an outlier which does not belong to this list. Is this statement possible given only mean and stddev?

Can I compute the probability with which this new value is part of the list? I'd like to have a probability as a result.

$\endgroup$
0
$\begingroup$

Absolutely. It is a known fact that for a sufficiently long list , (denoting mean by $\mu$ and standard deviation by $\sigma$) the range $[\mu-3\sigma,\mu+3\sigma]$ encompasses about (more than) $99.73\%$ of the data points, so if the new value is out of this range then it is $99.7\%$sure to be out of the list

You can somewhat use the concept of $p-value$ here. (Assuming the new value to follow gaussian distribution,since we don't know) ; find out the value of $\Phi(x)$--(CDF of $N(\mu,\sigma^2)|_{x=\text{new value}}$) Its $p-value=1-\Phi(x).$ If $p-value\lt $ some confidence level(say 0.05) then you can consider it within the list else not.

$\endgroup$
  • $\begingroup$ Ok, I should have worded more concrete: Can I compute the probability with which this new value is part of the list? I'd like to have a probability as a result. $\endgroup$ – Alfe Jun 27 '16 at 13:27
  • $\begingroup$ @Alfe I said so; that if the new value is within the range given then $P($it will belong to the list$)<1$. Google and you will find a derivation of how $99\%$ comes into play $\endgroup$ – Qwerty Jun 27 '16 at 13:30
  • $\begingroup$ @Alfe Take a look at the link $\endgroup$ – Qwerty Jun 27 '16 at 13:32
  • $\begingroup$ I took. So I think I need to compute Φ((v - μ) / σ), can you confirm this? I found that Φ(x) is not part of typical math computer libraries, but erf(x) is, and that Φ(x) = (1 + erf(x / sqrt(2))) / 2 . So the answer should be: p(v) = abs(Φ((v - μ) / σ) - Φ((μ - v) / σ)). Can you confirm this? $\endgroup$ – Alfe Jun 27 '16 at 13:56
  • 1
    $\begingroup$ given that this is posted to Mathematics, it feels to me like answers should err on the side of greater rigor ... the question might be better on CrossValidated $\endgroup$ – Ben Bolker Jun 27 '16 at 22:55
1
$\begingroup$

It is best to use a boxplot to find outliers. The problem with using the sample mean $\bar X$ and the sample SD $S$ is that an outlier seriously affects the values of $\bar X$ and $S$.

By contrast, the boxplot uses the median and the interquartile range to detect outliers. These measures of location and dispersion, respectively, are not much affected by outliers.

If you feel you must use $\bar X$ and $S$, then here is how to test observations one at a time for outliers: Omit the suspected outlier. Find $\bar X^*$ and $S^*$ from the remaining $n - 1$ observations. Then see if the omitted point is in some interval such as $(\bar X* - 2.5S^*, \bar X* - 2.5S^*)$. If so, the suspected observation is not judged an outlier. If outside the interval, then consider it an outlier. The disadvantage of this method is that you have to recompute $\bar X^*$ and $S^*$ afresh for each suspected outlier.

$\endgroup$
  • $\begingroup$ Yeah, well, in my usecase I need to characterize each of a large quantity of such lists with as little data as possible. Later then come values which are to be checked for fitting into their list or being a sign of an outlier. It seemed feasible to not store the complete lists but just their mean and SD. Your suggestion (omitting one point and computing the S again etc.) is (sadly) besides my situation. But thank you anyway :) $\endgroup$ – Alfe Jun 27 '16 at 21:46
0
$\begingroup$

Yes. You can use your Standard Deviation to tell you this. Think about what Standard Deviation is telling you.

$\endgroup$
  • $\begingroup$ Ok, I should have worded more concrete: Can I compute the probability with which this new value is part of the list? I'd like to have a probability as a result. $\endgroup$ – Alfe Jun 27 '16 at 13:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.