25
$\begingroup$

I'm self-studying machine learning and getting into the basics of linear regression models. From what I understand so far, a good regression model minimizes the sum of the squared differences between predicted values $h(x)$ and actual values $y$.

Something like the following:

$$\sum_{i=1}^m (h(x_i)-y_i)^2$$

Why do we square the differences? On one hand, it seems squaring them will allow us to get a positive number when the expected value is less than the actual value. But why can't this just be accounted for by taking the sum of the absolute values?

Like so:

$$\sum_{i=1}^m |h(x_i)-y_i|$$

$\endgroup$
  • 11
    $\begingroup$ The rationale I've always heard for using squares over absolute values is that squares are more friendly calculus-wise than absolute values. It's nasty to differentiate absolute values compared to squares, for example. $\endgroup$ – Clarinetist Oct 11 '14 at 14:16
  • 1
    $\begingroup$ We would love to have an easy solution for the absolute errors rather than squared errors (because squaring enhances the large errors). But the fact is that squared errors lead to a comfortably linear solution, not the absolute ones. $\endgroup$ – Yves Daoust Oct 11 '14 at 15:33
  • 1
    $\begingroup$ An interesting read on this topic ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=4775883&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D4775883 $\endgroup$ – Karthik Upadhya Oct 11 '14 at 16:35
  • 2
    $\begingroup$ Also: minimizing sum of squares penalizes large deviations much more than sum of absolute values. $\endgroup$ – Tyler Oct 11 '14 at 17:56
  • 1
    $\begingroup$ This (mitpress.mit.edu/books/introduction-machine-learning) is a good book for "Machine Learning". It includes lots of derivations instead of just giving the formula. $\endgroup$ – Nuri Tasdemir Oct 11 '14 at 23:58
22
$\begingroup$

Actually there are some great reasons which have nothing to do with whether this is easy to calculate. The first form is called least squares, and in a probabilistic setting there are several good theoretical justifications to use it. For example, if you assume you are performing this regression on variables with normally distributed error (which is a reasonable assumption in many cases), then the least squares form is the maximum likelihood estimator. There are several other important properties.

You can read some more here.

$\endgroup$
  • 2
    $\begingroup$ This is actually the more rigorous reason, as normal errors are everywhere. $\endgroup$ – Davidmh Oct 11 '14 at 21:40
  • 1
    $\begingroup$ A key feature of least squares (which a median-based approach lacks) is that it is unbiased, i.e., the sum of the errors is zero. By the Gauss-Markov Theorem, least-squares is the best linear unbiased estimator (BLUE). All that said, minimum absolute deviation (MAD), which is what is minimized under the second objective function, produces a robust estimate, which MSE does not. $\endgroup$ – Ghillie Dhu Oct 11 '14 at 22:57
10
$\begingroup$

If $h(x)$ is linear with respect to the parameters, the derivatives of the sum of squares leads to simple, explicit and direct solutions (immediate if you use matrix calculations).

This is not the case for the second objective function in your post. The problem becomes nonlinear with respect to the parameters and it is much more difficult to solve. But, it is doable (I would generate the starting guesses from the first objective function.

For illustration purposes, I generated a $10\times 10$ table for $$y=a+b\log(x_1)+c\sqrt{x_2}$$ ($x_1=1,2,\cdots,10$), ($x_2=1,2,\cdots,10$) and changed the values of $y$ using a random relative error between $-5$ and $5$%. The values used were $a=12.34$,$b=4.56$ and $c=7.89$.

Using the first objective function, the solution is immediate and leads to $a=12.180$, $b=4.738$,$c=7.956$.

Starting with these values as initial guesses for the second objective function (which, again, makes the problem nonlinear), it took to the solver $\Large 20$ iterations to get $a=11.832$, $b=4.968$,$c=8.046$. And all these painful iterations reduced the objective function from $95.60$ down to $94.07$ !

There are many other possible objective functions used in regression but the traditional sum of squared errors is the only one which leads to explicit solutions.

Added later

A very small problem that you could (should, if I may) exercise by hand : consider four data points $(1,4)$,$(2,11)$,$(3,14)$,$(4,21)$ and your model is simply $y=a x$ and your search for the best value of $a$ which minimizes either $$\Phi_1(a)=\sum_{i=1}^4 (y_i-a x_i)^2$$ or $$\Phi_2(a)=\sum_{i=1}^4 |y_i-a x_i|$$ Plot the values of $\Phi_1(a)$ and $\Phi_2(a)$ as a function of $a$ for $4 \leq a \leq 6$. For $\Phi_1(a)$, you will have a nice parabola (the minimum of which is easy to find) but for $\Phi_2(a)$ the plot shows a series of segments which then lead to discontinuous derivatives at thei intersections; this makes the problem much more difficult to solve.

$\endgroup$
4
$\begingroup$

One reason is the rest of calculations you need to do on the cost function.

For example to minimize the cost function (e.g. in gradient descent), you need to take derivations. Taking derivations of a squared value is much more easier than an absolute value.

$\endgroup$
4
$\begingroup$

Even when I'm not doing anything to do with differentiation, then I still like the squares. In fact, consider the $l_p$ norm, where you sum the $p$-powers, then raise to the power $1/p$: $$ || x ||_{l_p}^p = \sum_i |x|_i^p . $$ You can use this to "penalise" errors more aggressively. For example, when you square, it penalises more than just taking the absolute value: doubling the error quadruples it when squared. You can then let $p \rightarrow \infty $, and then this gives the $\sup$ norm: $$ || x ||_{l_\infty} = \sup_i x_i . $$ Alternatively, if doubling the error in your situation doesn't actually matter that much, you can choose smaller $p$, such as $p = 1/2$.

I hope this helps! :)

$\endgroup$
3
$\begingroup$

I study regression, and I used to wonder this very question myself.

Now I've come to the conclusion it's because of the geometry and linear algebra behind regression. Suppose we collect data on $n$ observations and to run a regression. When we minimize the sum of squared residuals, the way we do this (using Ordinary Least suares) is via projection matrices. We project a vector of explanatory variables (the "y" variables) onto a hyperplane of the explained variables (the "regressors" or "x" variables). By using projections, we are able to find the "closest" vector in the hyperplane (call it $\mathbf{x}\hat{\mathbf{\beta}}$, making the "error" vector of residuals $\hat{\mathbf{u}}$ as small as possible.

This is the key: when we choose $\hat{\mathbf{\beta}}$ to make the vector of residuals as "small" as possible, this means we are minimizing its Euclidean length: \begin{align} min |\mathbf{y}-\mathbf{x\beta}| &= min |\hat{\mathbf{u}}| \\ &= min \sqrt{\hat{\mathbf{u}}^\top\hat{\mathbf{u}}} \\ &=min \sqrt{\hat{u}_+\hat{u}_2+\cdots+\hat{u}_n} \end{align} And this is where the sum of SQUARES comes in. It's actually a geometric result.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy