Suppose we have some data points $\mathcal{D}=\{(x_i,y_i)\}_{0\le i\le N}$ and we would like to do some linear regression analysis on these points. Let's stick to two methods, least squares (i.e. minimising the L2 norm of the residues) and L1 regression (i.e. using L1 norm).

I have more often encountered the first rather than the latter. However the L1 method is much more rigid in the sense that it is less sensitive to outliers. So my question: L1 seems to be much better for linear regression, why would we care about L2? Some ideas/additional questions.

  • L2 problems seem to be easier to solve than L1. In any case, L2 would probably ask less computer power (is this difference in calculations needed significant for large datasets $\mathcal{D}$?)
  • Which method would be most used in an industry setting? Why? Or are these not used at all and does one use other methods?
  • Is there a situation where L2 regression is preferred over L1 regression (for reasons relying on the practical meaning of the data or problem, not computational ones)?
  • 1
    $\begingroup$ The main reason L2 is used much more than L1 is that more than 200 years ago, it was much easier and faster for Gauss to do least squares by hand than to solve a Linear Programming problem needed for L1. Tradition is a difficult habit to break, even when the original rationale no longer applies. $\endgroup$ – Mark L. Stone Oct 24 '20 at 15:40

The issues with L1 when compared with L2 are:

  • the former has a non differentiable points;
  • L1 has equal gradient everywhere which means that the loss will be the same if you are very far from the solution or very close;
  • L1 can have multiple solutions.

Also, L2 has an analytical solution so it is easy to implement.

You can also check Least_absolute_deviations compared to L2 regression on Wikipedia.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.