5
$\begingroup$

My understanding is that Ordinary Least Squares (Usually taught in Statistics classes) uses the vertical distance only when minimizing error/residuals (see Wikipedia for Ordinary Least Squares) with a modeled line. On the other hand, Linear Least Squares (Usually taught in Linear Algebra classes) uses vertical and horizontal distance components when minimizing the error/residuals (See Wikipedia for Linear Least Squares) with the modeled line, in effect minimizing the "closest" distance.

Is this correct?

Normally would one expect to get the same estimation of parameters for a linear model?

$\endgroup$
3
  • 1
    $\begingroup$ Not at all. Which of many linear models? What is the $purpose$ of the analysis? To use x to predict y, y to predict x? To see if dimensionality of x & y together can be reduced to z? To assess scatter about some hypothetical (possibly linear) relationship between x & y? $\endgroup$
    – BruceET
    Commented Oct 3, 2015 at 4:53
  • $\begingroup$ I'm assuming when you said "not at all" you were answering my second question. I was referring to a plain linear model of the form y = mx + b where m and x are being parameterized so that the distance between data-points and the line is minimal. $\endgroup$
    – WoodMath
    Commented Oct 3, 2015 at 18:54
  • $\begingroup$ Yes, sorry. Not at all for your second question. Regression line of x on y, regression line of y on x, and principle axis all have different estimated parameters for m, b, and variability about the line. $\endgroup$
    – BruceET
    Commented Oct 3, 2015 at 20:13

1 Answer 1

Reset to default
2
$\begingroup$

Ordinary Least Squares and Linear Least Squares are the same in the sense they minimize the vertical distance between the plane estimated and the measurements.
Yet, they have different assumption about the data:

  • Ordinary Least Squares (OLS) - In its stochastic model assumes IID white noise.
  • Linear Least Squares (LLS) - Allows white noise with different parameters per sample or correlated noise (Namely can have the form of Weighted Least squares).

Total Least Squares and PCA are the ones which minimize the "Shortest" distance (Perpendicular distance).

$\endgroup$
6
  • $\begingroup$ It's better that you say that OLS is linear least squares method. $\endgroup$
    – user168764
    Commented May 17, 2020 at 20:16
  • $\begingroup$ @Royi I disagree. They are not the same as far as I know. ordinary least squares assumes that errors in different observations are of the same order and no correlations are present in these errors. When for example the errors in y depend on the corresponding value of x, one must use weighted least-squares. Thus, ordinary and weighted are special cases of the linear least squares. $\endgroup$
    – Peaceful
    Commented Jan 18, 2021 at 7:37
  • $\begingroup$ @Peaceful, You won't find a uniformly accepted answer. But form my experience, everything in the form $ {\left\| A x - b \right\|}_{2}^{2} $ is considered ordinary or linear and on the other side there is non linear. Pay attention that LS might be used with no context of knowledge of the data (It is not, at least not necessarily, a stochastic method). Hence the reasons you give for weighted least squares are related to MLE with Gaussian Noise which falls back to a LS problem. $\endgroup$
    – Royi
    Commented Jan 18, 2021 at 9:26
  • $\begingroup$ @Royi Wikipedia has different pages for OLS and LLS. $\endgroup$
    – Peaceful
    Commented Jan 18, 2021 at 14:19
  • 1
    $\begingroup$ @Peaceful, I updated the answer. Thank You. $\endgroup$
    – Royi
    Commented Jan 19, 2021 at 10:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .