2
$\begingroup$

As the title says, I am interested in computing or approximating a percentile (e.g. 95%) for a large time interval (e.g., 1h) based on the percentiles of smaller time windows (30s). I am aware that the correct way would be to compute the percentile from scratch starting from the raw data. However, the data set is very large and would take way too much time (a 30s window can have 100k-100M data points).

To simplify things, assume that the distribution for all the time intervals are the same but the constants might be somewhat different. For example, values in a window follow a Poisson distribution but the parameters of the distribution might vary somewhat between windows. To put it in more visual terms, the shape of all the distributions is the same but there might be a "scaling" factor that changes.

I do not require a mathematical proof but some insight on what would be a reasonable way to proceed. Right now I am computing the median value of all the percentiles I am interested in. Could I improve on this?

$\endgroup$
0
$\begingroup$

If the rate is nearly the same, say $\lambda$ for a 30 second time interval, then the rate for a one hour time interval would be $120\lambda.$ So you would need to find the 95th percentile of $\mathsf{Pois}(100\lambda).$ If $\lambda$ itself is random (perhaps according to a gamma distribution), that might introduce another significant component of variability.

It might help toward a realistic model if you could give numbers of observed events in several typical 30-second intervals. Without any idea at all of the size or variability of the $\lambda$s, useful speculation even about approximate models is not possible. You say you have thousands or millions of 'records' every 30 seconds, but how many Poisson events do you typically have in a 30-second time period?

$\endgroup$
  • $\begingroup$ Thanks for the help! The events / records are IO latencies (response times) of fast solid-state drive(s). Each drive can sustain ~100k IO/s and there can be several of these drives. Plotting the response times, I think they resemble a shifted Poisson distribution with a small lambda -- most response times are around a certain value but there is a long tail. For a given dataset / experiment, the distribution type should be the same but there might be small changes over time in the shift of the distribution and in the tail length. $\endgroup$ – Radu Nov 9 '17 at 23:18
  • $\begingroup$ The number of events in a time window also stays roughly constant and I can compute it exactly from the average latency of each window. But I see now that the event count should be taken into consideration when computing the aggregate percentiles. $\endgroup$ – Radu Nov 9 '17 at 23:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.