I'm trying to find a few good algorithms which could solve the problem of finding anomalies in my data. One of the main problems is that I need to find it in the real time data.
So I came out with the idea of calculating average as my new data comes in and also calculating standard deviation. After I do that I add standard deviation to my average to get the
upper limit. And also take away standard deviation from the average to get
bottom limit. And then finally check if that value is outside the range.
That's a quick sketch:
/\ /\/\ / \ / \ ** IF VALUE IS ABOVE THEN ALERT ----/\----/----\/\----/\------/------\/\----- > upper limit ---/--\--/--------\--/--\----/----------\--/- > average --/----\/----------\/----\--/------------\/-- > bottom limit / \/ ** IF VALUE IS BELOW THEN ALERT
I used the approach from here.
The problems I am facing are:
so if I have a big spikes, let's say, every weekend I wouldn't like to get them as anomalies, because they happen every weekend.
But if I get something odd, even if it's not as big as the weekend's spikes, I'd like to get anomaly
also I'd like to have some other algorithm which will look on the volume on Monday this week and volume on Monday on the previous week and compare those values
going a bit more into details with the above I'm also trying to figure out how big the time block supposed to be:
- let's say that I check for volume daily, I could get volume of 46 the first day and 50 the next day which won't be odd. but if I would look at the second hour in the first day the count will be 3 and second hour in the second day 25 this is an anomaly I would like to know about
- how do I determine the appropriate frequency of measurement of events based on volume and variance of data?
These are the problems I would like to solve, I am not asking for the exact answer, but for pointing the algorithms which will be the best to solve each of these issues