I am responsible for metrics reporting within our IT organization. This involves monthly tracking of various metrics - for this post I will use one example: Incident Count (a.k.a Ticket Count). Each metric has a monthly count and a yearly avearage (the average of all of the months YTD). Here is a a real-life example:

Jan 2018: 4881

Feb 2018: 4502

Mar 2018: 5255

Apr 2018: 5310

May 2018: 5350

Jun 2018: 4576

Jul 2018: 4999

2018 YTD Monthly Avg: 4982

We want to add some up / down arrow indicators to show the trend direction. My boss has asked that these indicate some sort of significant trend. In other words, he doesn't want it to show that the trend is down if we have gone from 4,999 Incidents to 5,000 from one month to another. I thought about using a +/- standard deviation, maybe a certain percent, but I keep thinking there is a better way. I'd like to use the same methodology across all of our metrics, so the stats/math do not change, but work for metrics with any type of count or variance.

Some things that I can point out - since we are doing current year monthly numbers, our sample size is small (1-12 months), some metrics have larger numbers (4,000+) and others have smaller numbers (~100 or so). I feel like there should be a way to determine significant change (statistically) for a data set like this. Keep in mind, all of our metrics will be the same sample size (1-12 months). Are there any basic methods I can use that I can use to determine if my trend is truly going up or down in a significant way (other than just comparing two months). I am coding this up in R, and just want something standard I can use across the board and is reproducible. Thanks!

  • $\begingroup$ You might be interested in a control chart: en.wikipedia.org/wiki/Control_chart If you google "control chart in r" you will find information on R packages for control charts. $\endgroup$ – awkward Aug 7 '18 at 15:06
  • $\begingroup$ @awkward - Good stuff, I used this and I like it. I Think for now, to keep it simple, I'm going to use a % for all of my metrics so I can make it the same across the board, then as I move down the road, I can incorporate some more statistical elegance into this for each metric.... $\endgroup$ – azdatasci Aug 7 '18 at 16:45
  • $\begingroup$ Just a word of caution, it may not be good to base your control chart on percentages unless the control chart is designed for that purpose (in which case it may say something about "proportions"). Sometimes the control charts are designed to be used for "number of occurrences" of something, and if so that's how they should be used. $\endgroup$ – awkward Aug 7 '18 at 17:03
  • $\begingroup$ @awkward - Oh, just to clarify, I didn't mean I'd use it on % values, I meant, that for the time being, to determine a significant change, I'm going to look at the current and previous month's % difference. If it's +/- 3%, I'll just label it as an up or down trend. Anything inside of that will be considered no significant change. For now, this will work and I can take more time to wrap my head around using a real statistical method to determine significance. $\endgroup$ – azdatasci Aug 7 '18 at 17:36
  • $\begingroup$ OK, the important thing is that each type of control chart comes with a set of assumptions and limitations, and you want to use it within those constraints. $\endgroup$ – awkward Aug 7 '18 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.