# calculating least squares fit

Why do we use a Least Squares fit?

One answer by Chris Taylor begins with the assumption that we should look for

$$y_i=ax_i+e_i$$

This reference:

also supports Chris' choice and states that "We could measure distance from the points to the line horizontally, perpendicularly, or vertically. Since we oftentimes use regression to predict values of Y from observed values of X, we choose to measure the distance vertically."

But would it not be better to measure the 'perpendicular distance'? For example, if we assume that our 'fitted line' is as above, then the perpendicular distance, $\Delta$ from a predicted point $(x_i, y_i)$ to an actual data point $(x_0, y_0)$ would be $\Delta^2 = (x_i - x_0)^2 + (y_i - y_0)^2$

so the error, E, would be $$E = \sum_{i=1}^n \Delta_i$$ This is where I get a little confused. I know that

$$y_i=ax_i+e_i$$ $$x_i=(y_i-e_i)/a$$

so substituting for $(\Delta_i)^2$

$$E = \sum_{i=1}^n ((\frac{y_i-e_i}{a}) - x_0)^2 + ((ax_i+e_i) - y_0)^2$$

Then I differentiate with respect to a and set that derivative equal to 0? Not only am I getting lost here but my Latex skills are failing. The equation gets pretty complicated but it can be calculated. The question is, 'is it more accurate to use perpendicular distance rather than vertical distance'?

The reason for using vertical distance is that often you have a knob you turn that controls the important parameter of the experiment and then you measure the output. We believe you can set this parameter exactly (or so close that any error of the point is all in the measurement of the $y$ value, not the $x$ value). This is appropriate as long as the error in $x$ is small compared to the error in $y$ divided by $\frac {dy}{dx}$. This is often true, but not always.

• There are so many good answers here, I did not know who should actually get credit for the 'answer'. I marked Ross's answer as 'the' answer because of its clarity in translating the abstract precept to human terms. I understand now that x and y are not functionally equivalent and that one is an 'input' (controllable) where as the other is the 'output'. This sheds interesting light on machine learning in regression models. Perpendicular distance might be meaningful when we do not know how two variables are actually related - in other words- which is the input and which is the output. – aquagremlin Jun 13 '15 at 13:52
• btw, Ross, can you give an example of when the use of perpendicular distance would be preferred? See below discussion with Michael Hardy. I am amazed that both of you have such high reputation scores. But then again, the fact that you both addressed my obscure question only reflects your constant efforts. – aquagremlin Jun 14 '15 at 20:15

As said in previous answers, the problem depends on what is measured. Suppose that $x$ is with no error and $y$ is measured with some error. So, the vertical distance is the good choice.

For sure the problem can be different : suppose that for a know value of $x$ you measure $y$ (with some error) and also $z$ (with some error too) and, for any reason, you want to correlate $y$ to $z$. So, now, both independent and dependent variables are in error and the vertical distance is probably a good choice. However, you will need to know somehow the standard deviations associated to the $y_i$'s and to the $z_i$'s and this is what makes the problem often difficult.

If you ggogle for "orthogonal distance regression", "total least squares", you will find some interesting papers and discussions on this topic.

For the fit of a straight line through origin $(y=a\, x)$, the problem is not too bad and not complex. Consider a data point $(X_i,Y_i)$. It will be on the normal line to $y=a \,x$ and writing the normal as $$y=-\frac 1 a x+b$$ this gives $$Y_i=-\frac 1 a X_i+b$$ then the value of $b$. So, the two lines intersect at $$x_0=\frac{X_i+aY_i}{1+a^2} \ \ ,\ \ y_0=a\frac{X_i+aY_i}{1+a^2}$$ So, the square of the distance from the point $(X_i,Y_i)$ to the point$(x_0,y_0)$ is just $$d_i^2=\frac{(Y_i-a X_i)^2}{a^2+1}$$ and the problem is then to minimize $$\Phi(a)=\sum_{i=1}^n \frac{(Y_i-a X_i)^2}{a^2+1}$$ As usual, we take the derivative with respect to $a$ and after some algebra we arrive to a quadratic equation in $a$ $$\Big(\sum_{i=1}^n X_iY_i\Big)a^2+\Big(\sum_{i=1}^n (X_i^2-Y_i^2)\Big)a-\Big(\sum_{i=1}^n X_iY_i\Big)=0$$ which, for sure, is more complex than $$a=\frac{\sum_{i=1}^n X_iY_i}{\sum_{i=1}^n X_i^2}$$ given for the regression based on vertical distances.

For the fit of a straight line with intercept, the problem is already much more complex and the expressions of the parameters are not explicit.

1. To use the "nearest point on the line" involves drawing a perpendicular from the point to the line. It is more complicated than the expression in the OP.

2. There is a good reason apart from simplicity to use the vertical distance. In many circumstances, the data points are not just arbitrary scattered points in the plane. The x-coordinate represents known quantities, while the y-coordinate represents measured data (which may contain variability and error). For example, we measure the population of a city every year. The x-axis measures years. The vertical distance to the best-fit line is modeling deviation of the population, in each given year, from the model's prediction. If instead we used a diagonal best-fit line as proposed, there is no such natural explanation of what that line is modeling.

The $x$-variable may be in kilograms and the $y$-variable in dollars. If you change kilograms to grams or to metric tons, or change dollars to cents, then the lines that were perpendicular in the plane before, no longer are. No such problem afflicts conventional least-squares.

And suppose you classify people according to occupation (any of six types of jobs, say) and native language, in a community in which there are 10 of those. And you have a model that predicts income based on those. It says $$y_{ij} = \alpha+\beta_i+\gamma_j + \varepsilon_{ijk}$$ where $i$ is any of the six occupational classifications, $j$ is any of the native languages, and $\varepsilon_{ijk}$ appears in the $k$th obervation among all for whom the values of the classifications are $i,j$. You estimate $\alpha$, $\beta_i$, $\gamma_j$ based on a sample, using least squares. How do you measure "perpendicular distances" then?

• thank you for interesting counterexamples that highlight the importance of dimensional analysis. For example, one might suggest using orthogonal distance when the same dimension is being measured-for example, size of nose to size of genitals. But as you point out, the issue goes deeper. We are fundamentally regarding the two variables as different-one is being 'selected' and the other is being 'observed'. The selected variable is assumed to be 'controllable' and without error. – aquagremlin Jun 13 '15 at 14:04
• However, the language example you presented could be modeled with multiple dimensions (one variable for each language or occupation) The value of that variable would then reflect 'competence' or ability in that variable. The output (income) might then be more informative-it might show that income correlates with extreme competence in a variable. And not only a combination of variables which have only discrete binary states. – aquagremlin Jun 13 '15 at 14:07
• @aquagremlin : What you propose could be done only if the data on degree of language competence were available, but even then it doesn't bear on the point of the example. The point was that least squares is used in some cases where the proposed "perpendicular distance" makes no sense. ${}\qquad{}$ – Michael Hardy Jun 13 '15 at 17:31
• Thank you Michael. I do appreciate your input and perhaps I misunderstood what you said. Without being snarky or rude, I would like to clarify my understanding of what you said. Your example of language seems to be a classification rather than a regression problem. The binary nature of multiple inputs would not lend itself to a regression. The central part of my question is: in those cases where we would choose regression-how do we decide between 'vertical distance' and 'perpendicular distance', – aquagremlin Jun 14 '15 at 20:02
• The take-away from Ross's comment is that vertical distance is preferred when we can continuously control one variable.('turn the knob'). But what circumstances would encourage us to select perpendicular distance? I imagine that might be when we had data that was not 'selectable' and by that I am beginning to think that it is data that is not continuous. But I guess I should not put words in to Ross's mouth and just ask him for more clarification. – aquagremlin Jun 14 '15 at 20:05

Remember that regression are used for modelling, and thus it should not be reduced to a question of geometry.

given $x$, we wanna give a guess "$f(x)$" of what $Y$ will be. In our case our guess will be of the form $f(x)=ax+b$.

If we have observations $(x_1,y_1),\ldots,(x_n,y_n)$, we wanna measure how well our guess $ax+b$ fitted the data. A natural measure is the distance $r_i$ between what really happened $y_i$, and what we guessed it to be $f(x_i)$. That is $r_i=\vert y_i-(ax_i+b)\vert$. To get óne measure for all coordinates we sum all the differences(Residuals) into

$$F^*(a,b)=\sum_{i=1}^n r_i.=\sum_{i=1}^n \vert y_i-ax_i-b\vert$$ which we then wanna minimize in terms of $a,b$.

However, for various reasons we choose another intuitive "measure of good guess" $s_i=(y-ax_i-b)^2$ and minimize $$F(a,b)=\sum_{i=1}^n s_i$$