Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is there a formula that is capable of operating on streaming inputs and approximating standard deviation of the set of numbers?

share|cite|improve this question
up vote 1 down vote accepted

You can use the formula $\sigma = \sqrt{\bar{x^2}-(\bar x)^2}=\sqrt{\frac {\sum x^2}N-\left(\frac {\sum x}N\right)^2}$ Each sum can be accumulated as the data comes in. The disadvantage compared to averaging the data first and subtracting the average from each item is you are more prone to overflow and loss of significance, but mathematically it is equivalent.

share|cite|improve this answer

To add to the above, a more numerically stable approach (due to Knuth) is to keep track of $\sum (x-\bar{x})^2$ (called M2 in the algorithm below).

The following is copied from the following wikipedia page which is worth a read. https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance

def online_variance(data):
    n = 0
    mean = 0.0
    M2 = 0.0

    for x in data:
        n += 1
        delta = x - mean
        mean += delta/n
        M2 += delta*(x - mean)

    if n < 2:
        return float('nan')
    else:
        return M2 / (n - 1)
share|cite|improve this answer
1  
Though this may provide an answer, it is always good to include essential parts of the link in the body of the answer. Besides, this is an old question which has good accepted answer. So, unless you are contributing something new, it is always good to answer newer questions. – Shailesh May 3 at 5:22
    
While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review – Rise May 3 at 5:41
    
Cool, I editted the answer to include the relevant part of the wikipedia article. I think having a more numerically stable version of the algorithm easily available is definitely worth adding, especially since the algorithm is so simple. – ranga May 3 at 6:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.