0
$\begingroup$

For a long data series the overall Standard Deviation shall be collected. However, due to memory constraints the data has to be aggregated per day - in a way that only one number is stored per day (instead of the actual data series).

My current approach is

  • collect the variance of the data series per day (one number)
  • calculate the average of all variances
  • calculate square root of this average

if I compare this number to the actual Standard Deviaion over the complete data series over all days, I see that it is a reasonably good approximation of the Standard Deviation (usually below 1% off)

How do I calculate the precise deviation of the actual Standard Deviation against the approximation (ie the square root of average of aggregated variances)?

$\endgroup$

1 Answer 1

0
$\begingroup$

If you want to know the deviation precisely, you'll need to calculate both quantities precisely. You lose information about the data (a lot) when you save just the variance.

But you can get pretty close by looking at a pooled variance. The only additional information you'll need is the number of data points in the daily sample. (You might also save the mean of the data while you're at it.)

$\endgroup$
1
  • $\begingroup$ Indeed, pooled variance is the better approximation. Is there a general approach to calculate how far off an approximation is? I assume it also depends on the underlying data set, e.g. for normal distributed data even the average of the Standard Deviations per day appears to be sufficient. $\endgroup$ Commented Jan 22, 2014 at 22:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .