# Method of Least Squares-Why is it preferred? [duplicate]

Possible Duplicate:
Why do we use a Least Squares fit?

To find the normal equations for derivation of the regression line we use the method of least squares, We want to make the error smallest for each individual,so we may take summation of mod of the error term for each individual instead of taking the summation of the square of the errors of the individuals. Why is the former method used for regression and not the second one?

It is not true that min sum of absolute errors is never used. Least squares is used because it is equivalent to maximum likelihood when the model residuals are normally distributed with mean 0. But when the distribution of the error term is non-normal particularly if it has heavy tails the least squares estimates are not best and robust procedures such as minimum sum of absolute errors are preferable.

The real answer is that in many cases the least sum of squares of errors has a simple optimal solution whereas other loss functions have optimal solutions with a complicated form that need to be approximated numerically.

• This answer does not explain why the method of least squares is as popular as it is. – akkkk Sep 3 '12 at 16:38
• $x^2$ is differentiable with the smple derivative $2x$, whereas $|x|$ is not differentiable, thus depriving us of powerful methods. Also, the sums used in calculation (e.g. $sum x_iy_i$) are easy to compute by accumulation without keeping the individual values. Then again, just taking a simple algorithm cannot be the true reason - I prefer Michale Chernick's hint to the method just being the right one for normally distribued errors. – Hagen von Eitzen Sep 3 '12 at 17:36
• @HagenvonEitzen Considering that some fields settled on their standard statistical methods back when all calculations were done by hand, I find it plausible that ease of calculation was a sufficient reason. – octern Sep 3 '12 at 22:50

Because we prefer many small errors over one big error.

Physically, this is because a small amount of noise is present in any data, so a small deviation everywhere is only to be expected.

So technically, a regression analysis gives you a model of whatever model you already had, plus some noise on all data.

A big error at a specific point, however, indicates that the noiseless model does not really fit the data, since big errors are not typical for noise. This is why we fit against the assumption that big errors are unlikely.

• But in case of the former method there is also many small error and no big error – SN77 Sep 3 '12 at 16:30
• SN77: this is not true. If the errors are $1,1,1$ then you would have a total error of 3, but an equal total error for $0,0,3$. Least squares would say that individual errors of $1,1,1$ (total 3) is a better fit than $0,0,3$ (total 9). – akkkk Sep 3 '12 at 16:32
• But for 0,1,2 it would be 9 again. My question is in order to avoid big errors as you said if we take mod of the summation of errors then there is no possibility of positive and negative errors cancelling out the exact reason why we do not take summation of the errors. But for squares the deviations are also not cancelled out. In any case you are probably right but I am not getting your logic. – SN77 Sep 3 '12 at 16:39
• Please define "negative error", and in any case, clarify your question. – akkkk Sep 3 '12 at 16:40
• Say the regressed value be Y and the actual value be y. Now y-Y be be positive or negative, called positive and negative errors respectively. In case of ordinary summation these errors may cancel out. – SN77 Sep 3 '12 at 16:43

In addition to all the reasons given in the other answers, consider that in comparison to the sum of absolute values of errors, the sum of squared errors gives greater weight to large errors and less weight to small errors. This is because $$x^2 > |x| ~~ \text{if} ~~ 1 < |x| < \infty,\\ x^2 < |x| ~~ \text{if} ~~ 0 < |x| < 1.$$ This affects the fitting of the regression line in ways that some might consider more desirable.