Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I've been wondering for a while now if there's any deep mathematical or statistical significance to finding the line that minimizes the square of the errors between the line and the data points.

If we use a less common method like LAD, where we just consider the absolute deviation, then outliers make less difference to the final model, while if we take the cube of the error (or any other power higher than 2), then outliers are far more significant than with the least squares model.

I suppose what I'm really asking is mathematically, is raising the error to the power of 2 really that special. Is it say more "accurate" in some sense than raising the error to the power of 1.95 or 2.05???

Thanks!

share|improve this question
2  
Actually, in addition to LS and "least absolute deviation", there's also "Chebyshev fitting" or "minimax fitting", where you minimize the maximum of the errors between the line and the points. –  J. M. Sep 10 '11 at 1:37
1  
As for "why 2?": The 2-norm function has nice properties like differentiability. Also, it is the most appropriate norm to use when the errors in your $y$-coordinates are normally distributed... –  J. M. Sep 10 '11 at 1:38
2  
...and admittedly, people might use least squares when "least deviation" or minimax might have been more appropriate, simply because the algorithms for least squares (usually) require less computational effort. –  J. M. Sep 10 '11 at 1:45
    
And the most obvious reason is that, we don't want to consider the "sign of the error" when summing up the error terms. you can also take the 4th or any even number, or simply the infinity norm. –  user13838 Sep 10 '11 at 1:45
add comment

7 Answers 7

up vote 22 down vote accepted

Carl Gauss (the most famous person to live on earth in the 19th century, except for people who did not work in the physical and mathematical sciences) showed that least squares estimates coincide with maximum-likelihood estimates when one assumes independent normally distributed errors with 0 mean and equal variances.

share|improve this answer
1  
(+1) The connection to the normal distribution is the key. Regarding your parenthetical remark, while almost inarguably unrivaled as a mathematician, I think he has some solid competitors in the sciences at least in terms of name recognition in "popular" culture, both then and now (Darwin, Edison, Pasteur, Maxwell, etc.). –  cardinal Sep 10 '11 at 4:52
    
I said physical sciences, so maybe that rules out Darwin and Pasteur. Is Maxwell really more famous than Gauss? –  Michael Hardy Sep 10 '11 at 15:14
    
I know, I was just giving you a slight ribbing. Also, Pasteur is probably almost as well known as a chemist as he is as a microbiologist. Darwin I certainly give you. :) –  cardinal Sep 10 '11 at 15:53
1  
+1 the normal assumption underlying the error model is the key. Any computation advantage of least square is just by-product. Without normal assumption, lse is much less justified. However, with approximated normal error structure, for example t distribution with modest df, using least squares is still recommended. –  Yan Zhou Sep 11 '11 at 16:18
    
Thanks, this is just what I was looking for, something slightly deeper than the computational convenience. Do you know of any good texts that might explain these concepts, or at least give Gauss' argument? Thanks! –  tom Sep 12 '11 at 0:20
show 1 more comment

The answers provided to this point have considerable limitations and no answer in favor of squares is clear and convincing. Some are just dogmatic statements (squares are recommended). Others fail to relate the question to human life. In real life, we aspire to reduce errors, not their square. If squares are said to better predict the population, that is tautological if the standard of "prediction" uses squares rather than differences in its definition. What we need to know is whether squares predict differences better than differences predict differences. When two people predict the time outcomes of a race or the scores of a football match, who in their nutty mind would square their errors to compare accuracy? Appealing to "N dimensional space" does not address this. If we need to predict city water needs in multiple locations, and minimize errors, do you want to minimize the total water shortage, or for some odd reason treat differently the places that were poorly estimated (by squaring the errors)? that is to arbitrarily treat previous extreme observations differently that other cases in the study; if that is what you want to do, do so openly. Is there seriously real evidence that squares predict population differences better than differences predict population differences? That is what we need to know and that is what would answer the question. Gorand suggests that is not what Fisher estimated. Gorard notes that several fields are using least absolute regression instead of least squares. See Gorard, S. (2013) 'The possible advantages of the mean absolute deviation 'effect' size.', Social research update., 65 (Winter 2013). pp. 1-4. and Revisiting a 90-year-old debate: the advantages of the mean deviation. Stephen Gorard

share|improve this answer
add comment

Least squares fitting has the desirable property that if you have two different output values for the same input value, and you replace them with two copies of their mean, the least squares fit is unaffected. For example, the best fit line is the same for the following two sets of data:

0 1
0 5
1 5
2 6

and

0 3
0 3
1 5
2 6

If you use minimum-distance fitting, this is no longer the case.

share|improve this answer
add comment

Consider $n (x_1, x_2......x_n)$ measurements which have a normal distribution. Then the probability that only one of the measurements will occur is given by $P(X=xi) = e^{(x_i - x_T)^2/h^2}$ - where $x_T$ is the true value of the variable x to be measured.(I am ignoring the root pi factor here because it occurs for all the errors and is not central to our argument.). Now the probability that all of the measurements will occur in an experiment is given by the product: $P(X=x_1).P(X=x_2).P(X=x_3)....P(X=x_n)$- (assuming that the measurements are independent of each other - i.e the error from one measurement won't be carried over to the other - which would be the case if the experiment was well designed). The resulting probability is given by:

$P(x)$= $e^{(d_1^2 + d_2^2 + ......d_n^2)/h^2}$ where $d_i = x_i - x_T$

Now our aim is to find the value of $x$ for which the above probability is maximum. This would be the true value of the measurement.The above probability is maximum when, the exponent is minimum(remember the negative sign of the exponent) and the exponent is nothing but the sum of the squares of the deviations of the measurements from the true value.This is the idea behind sum of least squares.

We could use calculus: $dP(x)/dx = 0$

If you solve the resulting equation you find that the required value of x is $(x_1 + x_2 + x_3 +......x_n) / n$ - the arithemetic mean of the measurements. That's why we use the AM so much. In a reasonably well designed experiment, where the probability of small errors is large and of large errors is small and positive and negative errors occur with the same probablity (which is when you use a normal distribution), the AM is the most probable true value.

But remember that, the least squares method is only applicable when the measurements can be assumed to have a normal distribution. In other cases, the least squares method does not give the most probable true value.

share|improve this answer
add comment

16 months after asking the question, I've run into a different and quite physicsy answer, which I hope is useful to someone.

Suppose that in order to determine $m$ parameters of some model:

$$\text{Output} = f(\text{Inputs, parameters})$$ we have conducted $N>m$ experiments. We want to use the information from these experiments to best choose the $m$ parameters (so that the output of the model and the actual experimental value are as close together as possible).

Now comes the physicsy part: lets construct an N-dimensional phase space so that the $N$ (experimentally determined) outputs from our $N$ experiments are represented by a single point in this space (the Cartesian coordinates of this point are the outputs from each experiment). Call this the 'data-point'.

Secondly, if we choose an arbitrary set of parameters for our model, we can use the inputs for each experiment to construct a 'predicted output' for each experiment (by parsing the inputs through our model). There will be $N$ predicted outputs (one for each experiment) and these form a second point in the phase space, say the 'prediction-point'. As we varying the parameters this point moves about in an $m$-dimensional subspace of the phase space. And this is the important point:

The sum of the squares of the error terms (SSE) is the square of the distance between these two points in the $N$-dimensional phase space, just by Pythagoras' Theorem.

So minimising the sum of squares error is equivalent to minimising the distance between the data-point and the prediction-point in the $N$-dimensional phase space - a very natural way of calibrating our model.

Finally, from this Gauss' result makes some sense - if the data point is allowed to vary normally with 0 mean and equal variances, the error will be spherically-symmetric around the data-point, and so the closer our prediction-point is to the data-point in space, the better, and minimising this distance should give the maximum-likelihood estimator.

share|improve this answer
    
Thanks for this. After reading Michael Hardy's answer I didn't feel enlightened, and I wanted to ask him for elaboration. Your answer was exactly the elaboration I hoped to get. –  MJD Jan 24 '13 at 3:19
    
"So minimising the sum of squares error is equivalent to minimising the distance between the data-point and the prediction-point in the N-dimensional phase space - a very natural way of calibrating our model." Natural, yes, because we are used to reasoning in Euclidean spaces with the associated L2 (Euclidean) norm. There may be (perhaps non-intuitive or "unnatural") mathematical reasons for deviating from this norm towards others. –  lodhb Apr 14 '13 at 22:04
add comment

One very practical reason for using squared errors is that we are going to want to minimize the error, and minimizing a quadratic function is easy - you just differentiate it and set the derivatives to zero, which results in a linear equation - which we have centuries of tricks to help us solve.

I'll walk through a simple example: finding the best line through the origin that fits the data points $(y_i,x_i)$ for $i=1,\dots,n$. Our model for the data is

$$y_i = ax_i + \epsilon_i$$

where $\epsilon_i$ is the error of the approximation for the $i$th data point. Let's consider the error under what's technically called the $l^2$ norm, where we raise the absolute magnitude of each error to the power $2$ and then sum them:

$$S_2 = \sum_{i=1}^n |\epsilon_i|^2 = \sum_{i=1}^n (y_i - ax_i)^2$$

Minimizing this error with respect to $a$, we differentiate and set the derivative equal to zero:

$$\frac{\partial S_2}{\partial a} = -2\sum_{i=1}^n x_i (y_i - ax_i) = 0$$

and hence

$$\sum_{i=1}^n x_iy_i = a\sum_{i=1}^n x_i^2$$

so we recover the standard least-squares estimator of the slope,

$$a = \frac{\sum_{i=1}^n x_iy_i}{\sum_{i=1}^n x_i^2}$$

If we had used the $l^p$ norm for $p\neq 2$ instead of the $l^2$ norm, we would not have had an equation which was nearly so simple to solve. Indeed, minimizing anything other than the squared error function is generally only achievable with a numerical method.

share|improve this answer
add comment

Read section 5.14 Why Least Squares? of Meyer's Matrix Analysis and Applied Linear Algebra, which is available online).

share|improve this answer
1  
It might be better to link only to the book's web site, and not directly to the chapter's PDF. I feel that would better respect the author's wishes (regardless of how valid one considers those wishes to be). –  Rahul Sep 10 '11 at 3:30
    
@Rahul, fixed, thanks for the nudge. –  lhf Sep 10 '11 at 11:44
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.