0
$\begingroup$

We have a cumulative probability distribution function (cdf), we want to scale it down for using it in anomaly detection. The mapping should look like this.

CDF value: 0.1 ... 0.5 ... 0.9 ... 0.99 ... 0.999 ... 0.9999 ........

Mapped to: 0.001 ... 0.002 ... 0.01 ... 0.2 ... 0.3 ... 0.5 ........

So CDF greater than 0.9 is relevant and then the values should start increasing rapidly. The mapped scale is between 0 to 1, with 0.2 being strong anomaly and 1 being extreme anomaly.

Is there any standard function to scale CDF values, in above described manner?

Also is this a standard Statistics approach, or this approach is used in Anomaly Detection? (Any references will be helpful)

$\endgroup$
6
  • $\begingroup$ How about polynomal mappings? $x\mapsto x^2$ comes to mind as the most usual, but increasing the exponent will make it increasingly sharp. Your values are quite extreme so maybe $x^{20}$ provides a good scaling. $\endgroup$ – AlexR Mar 4 '15 at 23:00
  • $\begingroup$ It gives a good approximation but not exactly the same thing, as you can see 0.999 and 0.9999 have high difference in their mapping values. Using simple polynomial mappings, can't help it. $\endgroup$ – rg41 Mar 4 '15 at 23:04
  • $\begingroup$ Roots, maybe? Think of $1-\sqrt[n]{1-x}$ will not even be differentiable at $1$. $\endgroup$ – AlexR Mar 4 '15 at 23:06
  • $\begingroup$ This seems to be plausible solution (not exact kind of results though), but are there any standard approaches for this? Also, are there any references where people use this for anomaly detection? $\endgroup$ – rg41 Mar 4 '15 at 23:11
  • $\begingroup$ I guess most of the time you'll just highlight significance levels of a certain magnitude. Not an expert on the subject, though. $\endgroup$ – AlexR Mar 4 '15 at 23:12
0
$\begingroup$

As for your question about anomaly detection, you can use any threshold on CDF to declare an anomaly, despite what the CDF is, and you can apply any arbitrary non-negative increasing transformation to your CDF. So your proposed framework doesn't really add anything new to the concept of "anomaly detection." You're just changing around some thresholds for detection from one CDF to the next, leaving the set of finally declared "anomalies" intact. And frankly, 0.2 is quite an unusually low threshold to declare an anomaly anyway, at least in practice, so you might want a different pair of transformation/threshold assuming this is something you still want to do and sell to your audience.

$\endgroup$
3
  • $\begingroup$ This is just a part of problem statement, I am not adding anything new to anomaly detection. I can apply certain function such as root etc., but I want to know is there any standard approach for it, so that I can justify why I choose a certain function. $\endgroup$ – rg41 Mar 4 '15 at 23:20
  • $\begingroup$ @RamanGoyal What I'm saying is that you can use your original CDF for anomaly detection, with suitable threshold, without doing any transformation. Changing the CDF monotonically just changes what anomaly threshold you want, and doesn't "justify" the anomaly detection in any way unless you can show that you originally had a distorted CDF and can show how to "fix" it so that thresholds retain their intended meaning. CDF value of 0.99 means 1% chance of being occurring by chance, assuming your CDF is correct. $\endgroup$ – user2566092 Mar 4 '15 at 23:25
  • $\begingroup$ Well I need to change them because there are multiple attributes, and then I need to combine them. Also, the problem with using original CDF is that if value of 499 is 0.999, then value for 500 is 0.9999 and value for 500000 is 0.99999, what I essentially want is to distinguish them on a broader scale. $\endgroup$ – rg41 Mar 4 '15 at 23:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.