Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I have a set of data points. I extracted the Standard Deviation for the whole set, and also extracted STDs for a partition of the points where each element of the partition has at least two points. I averaged the subsets' STDs.

Of-course one can easily find examples where the averaged STDs are lower, equal, or greater than the 'full' STD. (for example, STD(1,1,11,11) is greater than average(STD(1,1), STD(11,11), and lower than average(STD(1,11), STD(1,11))).

However, I am interested in the relation between these two numbers. Is their expected value equal for random data? is there an equilibrium point? when does / doesn't averaging STDs 'make sense'?

share|improve this question
    
related: math.stackexchange.com/questions/223454 –  joriki Oct 30 '12 at 11:27

1 Answer 1

If the data points can be considered independent samples of a underlying random variable, if STD() is the unbiased standard deviation estimation operation, then the expected value of the STD() on any subset of the data points is equal to the standard deviation of the random variable itself, so the average of the STDs has the same expected value as the STD on the entire set.

In this case, averaging STDs should 'make sense' when the accuracy of the STD estimate obtained by averaging STDs on subsets is higher then the accuracy of a single STD estimation on the whole set.

Intuitively this is never the case, the accuracy of a single STD estimation on the whole available set is always the highest. Dividing a set in subsets is always possible and provides no additional information: if beneficial, it would have been incorporated in the formula for STD estimation.

I believe you can get to a proof by comparing the formula for the "variance of sample variance" http://en.wikipedia.org/wiki/Variance#Distribution_of_the_sample_variance for a whole set of N data points with the "variance of the average of sample variances" computed on M subsets of the original data set, i.e. 1/M times the "variance of sample variance" obtained on a subset of N/M data points.

Averaging STD could instead make sense when dealing with correlated samples, e.g. when you want to capture local statistical properties of a sampled random process.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.