0
$\begingroup$

I am working with a problem, where in I store the feature usages of any user on a daily bases. After each day the data gets accumulated and I wanted summarize the data and generate pre-calculated statistics from it.

What am I looking for:

A mathematical shortcut or a trick through which I can reproduce the series to calculate the Average & Median without actually storing the data for the last month

For example

A user Bob has used features Alpha and Beta in the past year as below:

alpha_bob = [40, 9823, 29840, 2380, 2397, 235, 4580, 2309, 2038, 2038, 95, 79230, 29370, 872, 2379, 2379, 239, 68623, 9723, 186, 2947, 23808, 2379, 39273, 975, 86238, 873, 72378]

beta_bob = [6272, 4980, 8776, 4394, 8013, 4461, 2135, 5360, 4291, 7587, 2517, 5196, 5585, 5347, 2244, 7531, 8341, 4723, 806, 506, 2055, 5787, 7833, 4651, 2487, 5268, 6935, 1328]

Another user Tom has used features Alpha and Beta in the past year as below:

alpha_tom = [4162, 2094, 233, 1357, 8141, 7754, 2754, 9828, 5708, 9713, 6038, 6395, 9378, 10, 3081, 1309, 8116, 9306, 4531, 8792, 7419, 2435, 5452, 1096, 2886, 4523, 1383, 376]

beta_tom = [4408, 2307, 8930, 4613, 9446, 9152, 5108, 8127, 5198, 3903, 4721, 1094, 4448, 1035, 5741, 8695, 5157, 7778, 7064, 9498, 6365, 7125, 9276, 4433, 9462, 1424, 339, 6114]

and I need to calculate 4 things from the data above:

  • usage of bob over the past 7, 14, 28 days (Median & Average)
  • usage of tom over the past 7, 14, 28 days (Median & Average)
  • usage of the feature alpha over the past 7, 14, 28 days (Median & Average)
  • usage of the feature beta over the past 7, 14, 28 days (Median & Average)

This is just sample data, the real data is enormous and I don't want to store the whole series for it all.

Is there any way I can get this data pre-calculated? Like calculating the Median for 27 days directly and then when the day is over I pick the current day usage and calculate the Median from the Median of 27 days and the current day.

I know it doesn't work like that, I am just trying to convey the intent in a more illustrative manner.

Also if I do have the pre-calculated 7, 14, 28 day stats, would I be able to calculate the usages for any other period, e.g. 21 days or 23 days per say (21 + 3), if yes how can I better store my data to reduce recalculating every-time

$\endgroup$
  • $\begingroup$ The median, I don't see a convenient way to do it, but the average you sure can but you will need to know the value from $7,14,28$ days ago as well. If you know that $\frac{x_1+x_2+\dots+x_{n}}{n} = \overline{X}$, if you know the value of $x_1$ and you know the value of $x_{n+1}$ then you can know $\frac{x_2+x_3+\dots+x_{n}+x_{n+1}}{n} = \overline{X}+\frac{x_{n+1}-x_1}{n}$. That reduces computation time, but still requires you keep track of the values. $\endgroup$ – JMoravitz Apr 22 at 19:07
  • $\begingroup$ In fact, by modifying what I said slightly, that should work as an argument that in order to calculate the "next averages" that you must have knowledge of the day which is removed. $\endgroup$ – JMoravitz Apr 22 at 19:09
  • $\begingroup$ @JMoravitz I see your point there, I need to maintain the information on the usages of the 29th day to effectively calculate the current day's average. is that what you mean? $\endgroup$ – iam.Carrot Apr 22 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.