# Why is there not a simpler way to calculate the standard deviation?

Steps of getting standard deviation. http://www.techbookreport.com/tutorials/stddev-30-secs.html:

1. Work out the average (mean value) of your set of numbers

2. Work out the difference between each number and the mean

3. Square the differences

4. Add up the square of all the differences

5. Divide this by one less than the number of numbers in your set - this is called the variance

6. Take the square root of the variance and you've got the standard deviation

Am I missing out something, or why do we need to square the differences in step 3?

Why not simply do a Abs (multiply all negative numbers by -1) in step 3?

Also, my second question is why do we need to divide by one less than the number of numbers in the set in step 5? why not simply divide by the number of numbers?

-
Overheard: "If the difference between $n$ and $n-1$ ever matters to you, then you are probably up to no good anyway; e.g. trying to substantiate a questionable hypothesis with marginal data." – J. M. Sep 3 '11 at 1:43

There is. Your alternative formulation of taking the absolute values of the differences instead of squaring them is called the mean absolute deviation (or average absolute deviation).

Both the mean absolute deviation and the standard deviation are used in practice, but much of the reason the standard deviation is more widely used is that it has nicer theoretical properties. For example, the mean and standard deviation are enough to specify which member of the family of normal distributions you are dealing with (edit: although this is convention, as Robert Israel notes in his comment below), and data values $x$ from a normal distribution with mean $\mu$ and standard deviation $\sigma$ can be transformed to data values $z$ from the standard normal distribution via $z = (x - \mu)/\sigma$. Another advantage of the standard deviation, as Robert Israel notes below, is that there is a simple formula for the standard deviation of the sum of independent random variables. (See also the paper referenced below for more on why we use the standard deviation, as well as some arguments in favor of the mean absolute deviation.)

For an answer to your second question, see my answer to "Sample Standard Deviation vs. Population Standard Deviation." In short, if you were calculating the standard deviation of a population rather than a sample, you would divide by the population size $n$. However, when you calculate the standard deviation of a sample, you have to estimate the population mean that would normally be in the formula with the sample mean. Doing so introduces a bias, as the data values tend to be slightly closer to the sample mean than to the population mean (as the sample mean is itself calculated from the data values). It turns out that dividing by $n-1$ rather than $n$ corrects that bias. (Proving that is a standard exercise in beginning statistical theory.)

Going back to your first question, I recent ran across the paper "Revisiting a 90-year-old debate: the advantages of the mean deviation," by Stephen Girard. The paper is worth reading in full, but let me summarize some of his main points.

Reasons for the standard deviation:

• It tends to have a smaller error, on average, when used to estimate a population standard deviation, and so is a more consistent estimate of the standard deviation of a population.
• The mean absolute deviation is much more difficult to manipulate algebraically. This makes developing more sophisticated analyses based on it more difficult.
• It's part of the definition of the widely-used normal distribution.
• Historical: Ronald Fisher, one of the leading figures in the development of statistics, championed its use.

Reasons for the mean absolute deviation:

• The standard deviation distorts the amount of dispersion (by the act of squaring the differences) in a data set.
• The mean absolute deviation tends to work better in the presence of errors in our data observations.
• The mean absolute deviation is less sensitive to outliers in the data (also because of the squaring in the standard deviation).
• It's simpler to understand if all you want is a quick measure of dispersion.
-
Although standard deviation is indeed used as one of the parameters of a normal distribution, this is only a convention, and you could equally well use the mean absolute deviation. In fact, the mean absolute deviation for a normal distribution of standard deviation $\sigma$ is $\sigma \sqrt{2/\pi}$. – Robert Israel Sep 1 '11 at 3:50
@Robert Israel: Thanks. I did not know that the mean absolute deviation for a normal distribution had such a simple form. – Mike Spivey Sep 1 '11 at 4:02
One of the main advantages of standard deviation, I think, is that there's a simple formula for the standard deviation of a sum of independent random variables. There is no such formula for mean absolute deviation. – Robert Israel Sep 1 '11 at 20:29
Shouldn't we not also note, that if we think in terms of variance, then the mean is the appropriate central value, and if we think in terms of absolute deviation the median is the most appropriate central value from which we determine the "centered" values of x? – Gottfried Helms Sep 1 '11 at 21:04
@leonbloy: Right, but isn't that basically what Gottfried Helms said? – Mike Spivey Sep 1 '11 at 22:11

Quantities are being squared, summed, and square rooted for the same reason that you do these things when finding the length of a vector. Just to pick a number, suppose that you have $10$ data points. Then before you took any measurements, there were ten variables $X_1, X_2, \ldots X_{10}$. Once you take the measurements, you have a vector in $\mathbb{R}^{10}$ denoted $\langle x_1,\ldots x_{10}\rangle$. and if you wanted its magnitude, you would sum squares and take a square root.

In the statistics setting, it is assumed that if you took measurements billions upon billions of times, there would be an average value $\mu$. If we knew $\mu$, we'd really be interested in studying all of the deviations $x_i-\mu$ in order to understand how individual values deviate from the mean. We'd study vectors of the form $\langle x_1-\mu,\ldots x_{10}-\mu\rangle$ instead of $\langle x_1,\ldots, x_{10}\rangle$. Unfortunately we do not know $\mu$. Instead, the average of the ten values (which we call $\bar{x}$) is taken to be an approximation of $\mu$, and we consider vectors of the form$\langle x_1-\bar{x},\ldots, x_{10}-\bar{x}\rangle$.

The size of these vectors would be found by summing squares and taking a square root. But why divide by $n-1$? It might seem to make more sense to divide by $n$. That would give a sense of finding the average displacement. However there is a historical reason for working this way. Once upon a time, integrals were approximated by hand and values were tabulated in the back of statistics books. If there were $10$ variables, it made sense to pretend ahead of time that you knew what $\bar{x}$ was and consider all the possible sets of measurements that would lead to that value of $\bar{x}$. This makes for a $9$-dimensional space. You have freedom to imagine $x_1$ through $x_9$, but then if $\bar{x}$ is fixed, you no longer have freedom for $x_{10}$ be whatever you like. (You may have heard the term "degrees of freedom" applied to this situation. In our example, there are 9 degrees of freedom.)

So it was actually a $9$-dimensional space that the old-timers did their calculus on. They did average the displacements, but only over the dimension of the sampling space, which was $9$, not $10$.

Another consideration: the variation in the sample data's deviations from $\bar{x}$ is a little larger than the variance would be from $\mu$ from sample to sample (since $\bar{x}$ itself varies). And dividing by $n-1$ instead of $n$ corrects somewhat for this added variation, giving a slightly larger value for the standard deviation than it would otherwise be.

I certainly haven't completely answered this, but I hope you can see the connection between statistics and vector calculus a little now. And for me, that connection is what explains the formula for standard deviation.

-
This is very near to that view on the question which I also take. I see the n measurements of one or more variable(s) as coordinates in a n -dimensional euclidean(rectangular) space. This also involves the question, whether the sampling did in fact measure at independent cases; if they are not independent of each other, the n-dimensional space is not orthogonal: so the independence/dependence of the individuals asked becomes a simple parameter of the model. The representations stddevs->length, mean->resultant (has least deviation), correlation->cos is then sequitur, obvious and coherent. – Gottfried Helms Sep 9 '11 at 6:43

Am I missing out something, or why do we need to square the differences in step 3? Why not simply do a Abs (multiply all negative numbers by -1) in step 3?

Historical tradition and theoretical convenience. There is nothing wrong with using absolute (unsquared) distance, or other measures of dispersion, such as Interquartile Range.

Also, my second question is why do we need to divide by one less than the number of numbers in the set in step 5? why not simply divide by the number of numbers?

There are some theoretical reasons for the traditional use of $n-1$ in the variance estimate from a sample, but in practice it is not important whether $n$ or $(n-1)$ is used. There is however an elegant theory of "degrees of freedom" that generalizes the $n-1$ correction to situations where it is more important, such as linear least-squares regression, Chi-squared tests, analysis of variance and other parts of basic statistics.

-

A lowbrow response (and an answer to the title question) is that there is an easier way to compute the standard deviation: as you look through your data, you compute three running totals:

1. How many items you've seen
2. The sum of all the items you've seen
3. The sum of the squares of all the items you've seen

From which you can compute $$\sigma = \sqrt{ \frac{1}{N-1} \left(\sum x^2 \right) - \frac{N}{N-1} \left(\frac{1}{N} \sum x \right)^2 }$$ or any of the other variations on you might want to use. You can compute the mean from this data too. If you keep sums of higher powers, there are additional useful statistics you can compute from the totals.

For some applications, the ability to compute statistics by keeping a running total in the above fashion is a huge, huge improvement over other approaches that require you to look through the data twice.

I'll leave the theoretical question of why the standard deviation is interesting and why you might want to use the sample standard deviation instead of the usual one to the other comments and answers that have already addressed it.

-
I don't recommend this formula for actual computational use: if the items are large and the variance is small, you stand to lose a lot of significant digits in the last subtraction (notwithstanding the fact that it might become something negative if the cancellation becomes perverse enough). – J. M. Sep 3 '11 at 1:40
Any sort of serious numeral algorithms have to pay attention to the pros and cons of alternatives. On this method specifically, if your numbers are integers, you can compute with no loss of precision at all. If you have a prior guess at the mean, you can tally $\sum (x-g)^2$, and derive a formula similar to the above. Hrm, now I'm curious about working out interesting one-pass methods for calculating the standard deviation.... – Hurkyl Sep 3 '11 at 7:25
Well yes, if you're working in exact arithmetic, this algorithm is fine. For the most part, I'm happy with the two-pass method. – J. M. Sep 3 '11 at 7:28