Why do we use a Least Squares fit?

Question

I've been wondering for a while now if there's any deep mathematical or statistical significance to finding the line that minimizes the square of the errors between the line and the data points.

If we use a less common method like LAD, where we just consider the absolute deviation, then outliers make less difference to the final model, while if we take the cube of the error (or any other power higher than 2), then outliers are far more significant than with the least squares model.

I suppose what I'm really asking is mathematically, is raising the error to the power of 2 really that special. Is it say more "accurate" in some sense than raising the error to the power of 1.95 or 2.05???

Thanks!

Answer 1

Carl Gauss (the most famous person to live on earth in the 19th century, except for people who did not work in the physical and mathematical sciences) showed that least squares estimates coincide with maximum-likelihood estimates when one assumes independent normally distributed errors with 0 mean and equal variances.

Answer 2

One very practical reason for using squared errors is that we are going to want to minimize the error, and minimizing a quadratic function is easy - you just differentiate it and set the derivatives to zero, which results in a linear equation - which we have centuries of tricks to help us solve.

I'll walk through a simple example: finding the best line through the origin that fits the data points $(y_i,x_i)$ for $i=1,\dots,n$. Our model for the data is

$$y_i = ax_i + \epsilon_i$$

where $\epsilon_i$ is the error of the approximation for the $i$th data point. Let's consider the error under what's technically called the $l^2$ norm, where we raise the absolute magnitude of each error to the power $2$ and then sum them:

$$S_2 = \sum_{i=1}^n |\epsilon_i|^2 = \sum_{i=1}^n (y_i - ax_i)^2$$

Minimizing this error with respect to $a$, we differentiate and set the derivative equal to zero:

$$\frac{\partial S_2}{\partial a} = -2\sum_{i=1}^n x_i (y_i - ax_i) = 0$$

and hence

$$\sum_{i=1}^n x_iy_i = a\sum_{i=1}^n x_i^2$$

so we recover the standard least-squares estimator of the slope,

$$a = \frac{\sum_{i=1}^n x_iy_i}{\sum_{i=1}^n x_i^2}$$

If we had used the $l^p$ norm for $p\neq 2$ instead of the $l^2$ norm, we would not have had an equation which was nearly so simple to solve. Indeed, minimizing anything other than the squared error function is generally only achievable with a numerical method.

Answer 3

Read section 5.14 Why Least Squares? of Meyer's Matrix Analysis and Applied Linear Algebra, which is available online).

Answer 4

16 months after asking the question, I've run into a different and quite physicsy answer, which I hope is useful to someone.

Suppose that in order to determine $m$ parameters of some model:

$$\text{Output} = f(\text{Inputs, parameters})$$ we have conducted $N>m$ experiments. We want to use the information from these experiments to best choose the $m$ parameters (so that the output of the model and the actual experimental value are as close together as possible).

Now comes the physicsy part: lets construct an N-dimensional phase space so that the $N$ (experimentally determined) outputs from our $N$ experiments are represented by a single point in this space (the Cartesian coordinates of this point are the outputs from each experiment). Call this the 'data-point'.

Secondly, if we choose an arbitrary set of parameters for our model, we can use the inputs for each experiment to construct a 'predicted output' for each experiment (by parsing the inputs through our model). There will be $N$ predicted outputs (one for each experiment) and these form a second point in the phase space, say the 'prediction-point'. As we varying the parameters this point moves about in an $m$-dimensional subspace of the phase space. And this is the important point:

The sum of the squares of the error terms (SSE) is the square of the distance between these two points in the $N$-dimensional phase space, just by Pythagoras' Theorem.

So minimising the sum of squares error is equivalent to minimising the distance between the data-point and the prediction-point in the $N$-dimensional phase space - a very natural way of calibrating our model.

Finally, from this Gauss' result makes some sense - if the data point is allowed to vary normally with 0 mean and equal variances, the error will be spherically-symmetric around the data-point, and so the closer our prediction-point is to the data-point in space, the better, and minimising this distance should give the maximum-likelihood estimator.

Answer 5

Consider n (x1, x2......xn) measurements which have a normal distribution. Then the probability that only one of the measurements will occur is given by P(X=xi) = e^-(xi - xT)^2/h^2 - where xT is the true value of the variable x to be measured.(I am ignoring the root pi factor here because it occurs for all the errors and is not central to our argument.). Now the probability that all of the measurements will occur in an experiment is given by the product P(X=x1).P(X=x2).P(X=x3)....P(X=xn)- (assuming that the measurements are independent of each other - i.e the error from one measurement won't be carried over to the other - which would be the case if the experiment was well designed). The resulting probability is given by:

P(x)= e^-(d1^2 + d2^2 + ......dn^2)/h^2 where di = xi - xT

Now our aim is to find the value of x for which the above probability is maximum. This would be the true value of the measurement.The above probability is maximum when, the exponent is minimum(remember the negative sign of the exponent) and the exponent is nothing but the sum of the squares of the deviations of the measurements from the true value.This is the idea behind sum of least squares.

We could use calculus: dP(x)/dx = 0

If you solve the resulting equation you find that the required value of x is x1 + x2 + x3 +......xn / n - the arithemetic mean of the measurements. That's why we use the AM so much. In a reasonably well designed experiment, where the probability of small errors is large and of large errors is small and positive and negative errors occur with the same probablity (which is when you use a normal distribution), the AM is the most probable true value.

But remember that, the least squares method is only applicable when the measurements can be assumed to have a normal distribution. In other cases, the least squares method does not give the most probable true value.