Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Let's take the numbers 0-10. Their mean is 5, and the individual deviations from 5 are
-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5
And so the average (magnitude of) deviation from the mean is $30/11 \approx 2.72$.

However, this is not the standard deviation. The standard deviation is $\sqrt{10} \approx 3.16$.

The first mean-deviation is a simpler and by far more intuitive definition of the "standard-deviation", so I'm sure it's the first definition statisticians worked with. However, for some reason they decided to adopt the second definition instead. What is the reasoning behind that decision?

share|improve this question
5  
Since you wonder why statisticians take the second definition why not ask them? In fact, see this question from the stats SE site: Standard deviation : Why square the difference instead of taking the absolute value? –  user116 Sep 16 '10 at 13:59
1  
I was browsing old questions and I came across this one. In my opinion, the best answer has been given at MathOverflow, and involves "the additivity of variance for independent variables and the central limit theorem": mathoverflow.net/questions/1048/… –  Rahul Oct 14 '10 at 19:44
    
Variances are additive! See my answer below. (Since there's already an "accepted answer", I fear mine my be missed if I don't mention this here.....) –  Michael Hardy Sep 17 '12 at 22:01

4 Answers 4

up vote 19 down vote accepted

Your guess is correct: least absolute deviations was the method tried first historically. The first to use it were astronomers who were attempting to combine observations subject to error. Boscovitch in 1755 published this method and a geometric solution. It was used later by Laplace in a 1789 work on geodesy. Laplace formulated the problem more mathematically and described an analytical solution.

Legendre appears to be the first to use least squares, doing so as early as 1798 for work in celestial mechanics. However, he supplied no probabilistic justification. A decade later, Gauss (in an 1809 treatise on celestial motion and conic sections) asserted axiomatically that the arithmetic mean was the best way to combine observations, invoked the maximum likelihood principle, and then showed that a probability distribution for which the likelihood is maximized at the mean must be proportional to $\exp(-x^2 / (2 \sigma^2))$ (now called a "Gaussian") where $\sigma$ quantifies the precision of the observations.

The likelihood (when the observations are statistically independent) is the product of these Gaussian terms which, due to the presence of the exponential, is most easily maximized by minimizing the negative of its logarithm. Up to an additive constant, the negative log of the product is the sum of the squares (all divided by a constant $2 \sigma^2$, which will not affect the minimization). Thus, even historically, the method of least squares is intimately tied up with likelihood calculations and averaging. There are plenty of other modern justifications for least squares, of course, but this derivation by Gauss--with the almost magical appearance of the Gaussian, which had first appeared some 70 years early in De Moivre's work on sums of Bernoulli variables (the Central Limit Theorem)--is memorable.

This story was researched, and is ably recounted, by Steven Stigler in his The History of Statistics - The Measurement of Uncertainty before 1900 (1986). Here I have merely given the highlights of parts of chapters 1 and 4.

share|improve this answer
    
Interesting, I never realized log-likelihood was Gauss-old. Thanks for the story, I'm going to have to check out the Stigler book in the library. –  J. M. Sep 16 '10 at 16:24
2  
This book is recommended in one of the statistics community wikis as an essential read. –  whuber Sep 16 '10 at 23:36

$\newcommand{\var}{\operatorname{var}}$ Variances are additive: for independent random variables $X_1,\ldots,X_n$, $$ \var(X_1+\cdots+X_n)=\var(X_1)+\cdots+\var(X_n). $$

Notice what this makes possible: Say I toss a fair coin 900 times. What's the probability that the number of heads I get is between 440 and 455 inclusive? Just find the expected number of heads ($450$), and the variance of the number of heads ($225=15^2$), then find the probability with a normal (or Gaussian) distribution with expectation $450$ and standard deviation $15$ is between $439.5$ and $455.5$. Abraham de Moivre did this with coin tosses in the 18th century, thereby first showing that the bell-shaped curve is worth something.

share|improve this answer

Variance is a natural measure of variability that comes up frequently in probability. Standard deviation, the square root of the variance, gives you a measure of dispersion that is on the same scale as the original data, which some may find more interpretable.

share|improve this answer
    
But that doesn't explain why standard deviation is better than mean absolute deviation. –  Michael Hardy Sep 18 '12 at 1:37
    
Hi @MichaelHardy! The variance comes up in various other settings, such the in the decomposition of the mean squared error and as a canonical parameter of the celebrated normal distribution. The link with the variance makes it a natural choice for a measure of dispersion that is on the same scale as the data. Thanks for your comment! –  user41583 Sep 18 '12 at 13:00

Squaring is nicer than taking the absolute value, e.g. it is smooth. It also leads to a definition of variance which has nice mathematical properties, e.g. it is additive. But for me the theorem that really justifies using standard deviation over the mean absolute error is the central limit theorem. The central limit theorem is at work whenever we measure the mean and standard deviation of a distribution we assume to be normal (e.g. heights in a population) and use that to make predictions about the entire distribution, since a normal distribution is completely specified by its mean and standard deviation.

share|improve this answer
    
Isn't a normal distribution also completely specified by the mean and mean absolute error? I.E. for normal distributions there is a bijection between s.d. and mean error. –  Oscar Cunningham Sep 16 '10 at 15:06
1  
Sure, but the math is easier with standard deviation (e.g. the probability density function is easier to write down). Let me also quote from the Wikipedia article: "The Gaussian distribution was named after Carl Friedrich Gauss, who introduced it in 1809 as a way of rationalizing the method of least squares." –  Qiaochu Yuan Sep 16 '10 at 15:52
1  
@Qiaochu: The Wikipedia article may be a little off. The Gaussian density appears to have been derived as a way of justifying the use of the arithmetic mean (rather than, e.g., the median) as a way to combine observations subject to independent random error. Least squares drops out from that as a byproduct (in the same way that the median, least absolute values, and Laplace's double exponential distribution are connected). –  whuber Sep 16 '10 at 16:18
5  
@Qiaochu: Plenty of distributions are specified completely by a small number of parameters, such as a mean and SD. (Take any positive measure with finite second moment, then normalize it, translate, and rescale it to create a family of such distributions.) The import of the CLT is that under mild conditions a sum of variables governed by almost any distribution whatsoever, even distinctly non-Normal ones, converges (often rapidly) to a Normal distribution as the number of terms in the sum grows large. (The CLT is of no interest for sums of Normals, because they are automatically Normal.) –  whuber Sep 16 '10 at 16:23
3  
@whuber: If you are certain wikipedia is a little off (and you have references to back that up), please be sure to edit the article to correct the mistake! :) –  BlueRaja - Danny Pflughoeft Sep 16 '10 at 17:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.