Reading about variance and it occurred to me that this squaring business seems to be used many places in statistics.

I think I understand that the square is used to help "weight" values which are further from the mean more heavily, so they don't get swallowed up in when there are a lot of less-deviant values? but, why was squaring chosen, and not cubing? Are there situations where it would make more sense to cube the deviations and not square them? Or was squaring just chosen because it made the deviations powerful, but not too powerful?

  • $\begingroup$ Related: Motivation behind standard deviation? That question asks why square the deviations instead of taking their absolute value, but the answers are equally valid for this question. $\endgroup$ – Rahul Sep 19 '12 at 0:09

The squared deviations are used in regression because if the error terms in the model are independent and identically distributed normal random variables with zero mean minimizing squared error is the same as maximizing the likelihood which is an optimal estimation procedure.

However when the error terms are not normally distributed the least squares estimates usually are poor because they are too highly influenced by ponts that deviate largely from the line that would fit the other points. So rether than go to cubing the errors the sum of the absoulte value of the error terms is what is minimized. Other robust options give varying weights depending on the location of the points.

  • $\begingroup$ Interesting. Why is minimizing a squared error the same as maximizing likelihood? What's the mathematical relationship? $\endgroup$ – paIncrease Sep 19 '12 at 0:13
  • $\begingroup$ oh, estimator quality/efficiency is quantified by the mean squared error. $\endgroup$ – paIncrease Sep 19 '12 at 0:17
  • $\begingroup$ but why were squares chosen? I don't see anything in en.wikipedia.org/wiki/Maximum_likelihood that explains this aside from saying that's how you gauge efficiency. $\endgroup$ – paIncrease Sep 19 '12 at 0:18
  • 1
    $\begingroup$ Consider simple linear regression where y$_i$=ax$_i$ + b + e$_i$ for i=1,2,..,n and the e$_i$ are N(0, σ$^2$) then the likelihood is П$_i$φ(e$_i$) where φ is a normal density with mean 0 and variance σ$^2$. The density has the terms exp(-e$_i$$^2$/(2 σ$^2$) for each i. The product of the exponentials is then equal to exp[-Σe$_i$$^2$/(2 σ$^2$)] This term is the only term involving the parameters a and b. So maximizing the likelihood requires minimizing the negative exponent of this exponential. But that means minimizing the sum of squared residuals. $\endgroup$ – Michael Chernick Sep 19 '12 at 3:25

I wrote the following answer to another question quite recently (link) and it seems relevant here too:

$\newcommand{\var}{\operatorname{var}}$ Variances are additive: for independent random variables $X_1,\ldots,X_n$, $$ \var(X_1+\cdots+X_n)=\var(X_1)+\cdots+\var(X_n). $$

Notice what this makes possible: Say I toss a fair coin 900 times. What's the probability that the number of heads I get is between 440 and 455 inclusive? Just find the expected number of heads ($450$), and the variance of the number of heads ($225=15^2$), then find the probability with a normal (or Gaussian) distribution with expectation $450$ and standard deviation $15$ is between $439.5$ and $455.5$. Abraham de Moivre did this with coin tosses in the 18th century, thereby first showing that the bell-shaped curve is worth something.

end of quote

There's more to the use of squares in statistics than that; there's also the whole topic of analysis of variance, where one decomposes sums of squares. Michael Chernick's answer addresses another aspect.

As for cubes, there are occasions to consider the sum of cubes of deviations from the mean: those are involved in skewness of distributions. But they don't work as a measure of dispersion. Here's a fact that seems not to be widely known; it's a simple exercise to prove it: Averages of cubes of deviations from the mean are also additive. (This doesn't work for $4$th powers or higher powers.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.