Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Let's take the numbers 0-10. Their mean is 5, and the individual deviations from 5 are
-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5
And so the average (magnitude of) deviation from the mean is $30/11 \approx 2.72$.

However, this is not the standard deviation. The standard deviation is $\sqrt{10} \approx 3.16$.

The first mean-deviation is a simpler and by far more intuitive definition of the "standard-deviation", so I'm sure it's the first definition statisticians worked with. However, for some reason they decided to adopt the second definition instead. What is the reasoning behind that decision?

share|cite|improve this question
5  
Since you wonder why statisticians take the second definition why not ask them? In fact, see this question from the stats SE site: Standard deviation : Why square the difference instead of taking the absolute value? – user116 Sep 16 '10 at 13:59
1  
I was browsing old questions and I came across this one. In my opinion, the best answer has been given at MathOverflow, and involves "the additivity of variance for independent variables and the central limit theorem": mathoverflow.net/questions/1048/… – Rahul Oct 14 '10 at 19:44
    
Variances are additive! See my answer below. (Since there's already an "accepted answer", I fear mine my be missed if I don't mention this here.....) – Michael Hardy Sep 17 '12 at 22:01
up vote 26 down vote accepted

Your guess is correct: least absolute deviations was the method tried first historically. The first to use it were astronomers who were attempting to combine observations subject to error. Boscovitch in 1755 published this method and a geometric solution. It was used later by Laplace in a 1789 work on geodesy. Laplace formulated the problem more mathematically and described an analytical solution.

Legendre appears to be the first to use least squares, doing so as early as 1798 for work in celestial mechanics. However, he supplied no probabilistic justification. A decade later, Gauss (in an 1809 treatise on celestial motion and conic sections) asserted axiomatically that the arithmetic mean was the best way to combine observations, invoked the maximum likelihood principle, and then showed that a probability distribution for which the likelihood is maximized at the mean must be proportional to $\exp(-x^2 / (2 \sigma^2))$ (now called a "Gaussian") where $\sigma$ quantifies the precision of the observations.

The likelihood (when the observations are statistically independent) is the product of these Gaussian terms which, due to the presence of the exponential, is most easily maximized by minimizing the negative of its logarithm. Up to an additive constant, the negative log of the product is the sum of the squares (all divided by a constant $2 \sigma^2$, which will not affect the minimization). Thus, even historically, the method of least squares is intimately tied up with likelihood calculations and averaging. There are plenty of other modern justifications for least squares, of course, but this derivation by Gauss--with the almost magical appearance of the Gaussian, which had first appeared some 70 years early in De Moivre's work on sums of Bernoulli variables (the Central Limit Theorem)--is memorable.

This story was researched, and is ably recounted, by Steven Stigler in his The History of Statistics - The Measurement of Uncertainty before 1900 (1986). Here I have merely given the highlights of parts of chapters 1 and 4.

share|cite|improve this answer
    
Interesting, I never realized log-likelihood was Gauss-old. Thanks for the story, I'm going to have to check out the Stigler book in the library. – J. M. Sep 16 '10 at 16:24
2  
This book is recommended in one of the statistics community wikis as an essential read. – whuber Sep 16 '10 at 23:36

Squaring is nicer than taking the absolute value, e.g. it is smooth. It also leads to a definition of variance which has nice mathematical properties, e.g. it is additive. But for me the theorem that really justifies using standard deviation over the mean absolute error is the central limit theorem. The central limit theorem is at work whenever we measure the mean and standard deviation of a distribution we assume to be normal (e.g. heights in a population) and use that to make predictions about the entire distribution, since a normal distribution is completely specified by its mean and standard deviation.

share|cite|improve this answer
1  
Isn't a normal distribution also completely specified by the mean and mean absolute error? I.E. for normal distributions there is a bijection between s.d. and mean error. – Oscar Cunningham Sep 16 '10 at 15:06
1  
Sure, but the math is easier with standard deviation (e.g. the probability density function is easier to write down). Let me also quote from the Wikipedia article: "The Gaussian distribution was named after Carl Friedrich Gauss, who introduced it in 1809 as a way of rationalizing the method of least squares." – Qiaochu Yuan Sep 16 '10 at 15:52
1  
@Qiaochu: The Wikipedia article may be a little off. The Gaussian density appears to have been derived as a way of justifying the use of the arithmetic mean (rather than, e.g., the median) as a way to combine observations subject to independent random error. Least squares drops out from that as a byproduct (in the same way that the median, least absolute values, and Laplace's double exponential distribution are connected). – whuber Sep 16 '10 at 16:18
5  
@Qiaochu: Plenty of distributions are specified completely by a small number of parameters, such as a mean and SD. (Take any positive measure with finite second moment, then normalize it, translate, and rescale it to create a family of such distributions.) The import of the CLT is that under mild conditions a sum of variables governed by almost any distribution whatsoever, even distinctly non-Normal ones, converges (often rapidly) to a Normal distribution as the number of terms in the sum grows large. (The CLT is of no interest for sums of Normals, because they are automatically Normal.) – whuber Sep 16 '10 at 16:23
3  
@whuber: If you are certain wikipedia is a little off (and you have references to back that up), please be sure to edit the article to correct the mistake! :) – BlueRaja - Danny Pflughoeft Sep 16 '10 at 17:20

Variance is a natural measure of variability that comes up frequently in probability. Standard deviation, the square root of the variance, gives you a measure of dispersion that is on the same scale as the original data, which some may find more interpretable.

share|cite|improve this answer
    
But that doesn't explain why standard deviation is better than mean absolute deviation. – Michael Hardy Sep 18 '12 at 1:37
    
Hi @MichaelHardy! The variance comes up in various other settings, such the in the decomposition of the mean squared error and as a canonical parameter of the celebrated normal distribution. The link with the variance makes it a natural choice for a measure of dispersion that is on the same scale as the data. Thanks for your comment! – user41583 Sep 18 '12 at 13:00

$\newcommand{\var}{\operatorname{var}}$ Variances are additive: for independent random variables $X_1,\ldots,X_n$, $$ \var(X_1+\cdots+X_n)=\var(X_1)+\cdots+\var(X_n). $$

Notice what this makes possible: Say I toss a fair coin 900 times. What's the probability that the number of heads I get is between 440 and 455 inclusive? Just find the expected number of heads ($450$), and the variance of the number of heads ($225=15^2$), then find the probability with a normal (or Gaussian) distribution with expectation $450$ and standard deviation $15$ is between $439.5$ and $455.5$. Abraham de Moivre did this with coin tosses in the 18th century, thereby first showing that the bell-shaped curve is worth something.

share|cite|improve this answer

This answer is merely a summary of Stephen Gorard (2005), "Revisiting a 90-Year-Old Debate: The Advantages of the Mean Deviation" (PDF), which argues that we should abandon the standard deviation (SD) in favor of the mean deviation (MD).

The apparent superiority of SD [standard deviation] is not as clearly settled as is usually portrayed in texts

This "apparent superiority" stems from the following:

(1) SD is more efficient than MD under ideal circumstances

(2) it is easier to manipulate algebraically

(3) SD has now become a tradition, and much of the rest of the theory of statistical analysis rests on it

Regarding (1), Eddington (1914) had first shown that MD was empirically superior to SD. Fisher (1920) then countered that

that SD was more efficient than MD under ideal circumstances, and many commentators now accept that Fisher provided a complete defence of the use of SD.

But

The mean deviation is actually more efficient than the standard deviation in the realistic situation where some of the measurements are in error, more efficient for distributions other than perfect normal, closely related to a number of other useful analytical techniques, and easier to understand.

Another argument against SD is that

The act of squaring makes each unit of distance from the mean exponentially (rather than additively) greater, and the act of square-rooting the sum of squares does not completely eliminate this bias. ... our use of SD rather than MD forms part of the pressure on analysts to ignore any extreme values


Another proponent of the MD over the SD is Nassim Nicholas Taleb (2014), who claims that

It is all due to a historical accident: in 1893, the great Karl Pearson introduced the term "standard deviation" for what had been known as "root mean square error". The confusion started then: people thought it meant mean deviation. The idea stuck: every time a newspaper has attempted to clarify the concept of market "volatility", it defined it verbally as mean deviation yet produced the numerical measure of the (higher) standard deviation.

share|cite|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.