I am watching an online Machine Learning course and one of the first topics is Linear Regression. I got the intuition behind the Cost function but I don't understand why the formula sums the squared difference and not the just the difference between the hypothesis and y. Can someone explain to me why the formula is: Cost Function

  • 1
    $\begingroup$ Using a squared distance makes the objective function differentiable, which can be helpful. Also the given penalty function penalizes large errors more severely than an $L_1$ penalty. $\endgroup$ – littleO Mar 25 '17 at 14:46
  • $\begingroup$ @littleO - So why stop at power $2$? Let's take a power of $20000000$: clearly differentiable, and penalizes even more severely! $\endgroup$ – uniquesolution Mar 25 '17 at 15:46

Well, the short answer is that it doesn't have to be squared. For instance, it can be any loss function, or any metric based on the $p$-norm: $$ ||x-y||_p = \left[ \sum_i |x_i - y_i|^p \right]^{1/p} $$ with $||d||_\infty=\max_i d_i$. See also $L_p$ space. The case you are considering is $||d||_2^2$.

Note obviously though that the difference $h_\theta(x^{(i)}) - y^{(i)}$ is a bad choice since sometimes the difference will be negative and other times positive so summing them wouldn't make any sense; using $|h_\theta(x^{(i)}) - y^{(i)}|$ or $(h_\theta(x^{(i)}) - y^{(i)})^2$ are the most obvious fixes to this. Hopefully you already knew this though.

I think there are several reasons specific to this case, however, that squared error is used for linear regression:

  1. As one commenter mentioned, using $||\cdot||_2^2$ is differentiable (unlike $p=1$ or $p=\infty$), meaning e.g. gradient descent is easy to implement (and in this case, convex).

  2. It is computationally efficient.

  3. Using the squared difference penalizes outliers more than using the absolute difference, but less than higher $p$ values. In a sense, it strikes a balance. As $p$ increases, the ability for a single outlier point to affect the entire model becomes larger and larger (which is often not desirable). Generally speaking, deciding the optimal $p\in\mathbb{R}_{\geq 1}$ might be a very hard problem in itself (although not often considered for linear regression specifically I suppose), so most people stick to $p=1,2,\infty$. The squared difference is a nice middle ground, in other words (but note that other metrics are more common elsewhere).

  4. The squared difference is very well-studied and understood in statistics. It is so common it is literally called ordinary least squares (OLS). This means that there are certain mathematical facts about the model that are already known, which can be utilized immediately. For instance, under some basic assumptions on the noise, the resulting parameters are the best linear unbiased estimators and/or the minimum-variance mean-unbiased estimators. We also know the exact distribution of the estimates of the coefficients from OLS, which can be useful for determining the reliability of the model or feature selection in higher dimensions, for instance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.