I've been wondering for a while now if there's any deep mathematical or statistical significance to finding the line that minimizes the square of the errors between the line and the data points.
If we use a less common method like LAD, where we just consider the absolute deviation, then outliers make less difference to the final model, while if we take the cube of the error (or any other power higher than 2), then outliers are far more significant than with the least squares model.
I suppose what I'm really asking is mathematically, is raising the error to the power of 2 really that special. Is it say more "accurate" in some sense than raising the error to the power of 1.95 or 2.05???