# Simple Least-Squares Regression Question

Given a set of 5 points (i.e. (1, 3), (2, 8) etc...), how can I get just the slope of the best fit line?

I've been looking up least squares regression, but I'm rather statistics ignorant and don't understand most of the terminology and math behind it. Can anyone explain it a bit more simply?

-
It's a bit hard to construct a good answer because one has to guess more simply than what. Could you perhaps point to one of the resources you fail to understand, such as to give an upper bound for the level of answer you desire? It would also help if you could sketch, in a few sentences, the most advanced facts you already know about the problem, such that we don't have to waste focus explaining that again unnecessarily. For example, do you know what distinguishes a least-squares fit from other possible fits and why it's the one you want? –  Henning Makholm Nov 15 '11 at 20:07

By best-fit line, I presume you mean the least-squares fit. The "least-squares fit line" for the given data $\{ (x_i, y_i) \}_{i=1}^n$ is, by definition, simply the line $\ell_{a,b}$ with the equation $y = a+bx$ that minimizes the least-square error: $$Q(a,b) := \sum_{i=1}^n (y_i - a - bx_i)^2.$$ Notice that the quantity $|y_i - a- bx_i|$ is a measure of the deviation of the point $(x_i, y_i)$ from the line. Squared error refers to the fact that we are summing (over the $n$ data points) the sum of squares of these deviations from the line. [Another reasonable choice could be to minimize the sum of errors $\sum\limits_{i=1}^n \ |y_i - a - bx_i|$, but least squares has the advantage that it is easy to compute the minimizer analytically*.]

To find the line $\ell_{a,b}$ that minimizes $Q$, we resort to calculus. Taking partial derivatives of $Q$ w.r.t. $a$ and $b$, we get: $$\begin{eqnarray*} \frac{\partial Q}{\partial a} &=& \sum_{i=1}^n 2 (a + bx_i - y_i) = 2an + 2b \sum_i x_i - 2\sum_i y_i. \\ \frac{\partial Q}{\partial b} &=& \sum_{i=1}^n 2 (a + bx_i - y_i) x_i = 2a \sum_i x_i + 2b \sum_i x_i^2 - 2\sum_i x_i y_i. \end{eqnarray*}$$ Setting both the partial derivatives to $0$, you can solve for $a$ and $b$.

-
Why do we consider "least squares", you may ask? It boils down to making two (not often reasonable) assumptions about the data you have: 1) the errors in your data are only in your ordinates $y_i$ (the abscissas $x_i$ are error-free), and 2) the errors are normally distributed. If even the abscissas are contaminated with error, then one should consider other methods like orthogonal regression. –  Guess who it is. Nov 16 '11 at 9:02
Thank you this helps a lot! –  pdeuchler Nov 17 '11 at 5:25

If you have $(x_1, y_1), ..., (x_n, y_n)$ and all you care to do is fit a straight line to the data (ignoring any actual sorts of statistics) a reasonable thing to do is minimize $$\sum_{i = 1} ^ n (y_i - \alpha - \beta x_i)^2$$ with respect to $\alpha$ and $\beta$, with solution say $\hat \alpha$ and $\hat \beta$; the line $\ell(x) = \hat \alpha + \hat \beta x$ is the line corresponding to your so-called "least squares" fit. Why is this reasonable? Well, the value $y_i - \alpha - \beta x_i$ is the amount that our line has missed the value of $y_i$ by. We would like to construct a line that, in general, doesn't miss by much. So we aim to minimize the above sum. We square this difference to make things positive since we don't want negative and positive misses to cancel each other out; we care about the magnitude of the miss, not the sign. We also could have minimized $\sum |y_i - \alpha - \beta x_i|$ - this gives a different fit with somewhat different properties, but in general this fit is more difficult to do since we can't differentiate $|\cdot|$.

The slope of this line is the value $\hat \beta$. To get $\hat \beta$ set $\bar y = n^{-1} \sum y_i$, and $\bar x = n^{-1} \sum x_i$. It turns out that the value of $\hat \beta$ is given by $$\frac{\sum (x_i - \bar x)(y_i - \bar y)}{\sum (x_i - \bar x)^2}$$

which you can get by differentiating $f(\alpha, \beta) = \sum (y_i - \alpha - \beta x_i)^2$ with respect to $\alpha$ and $\beta$.

-
Another problem with minimizing $\sum |y_i - \alpha - \beta x_i|$ is that it very often leads to underconstrained solutions where the "best fit" has quite a bit of play -- it can wiggle back and forth without changing $\sum |y_i - \alpha - \beta x_i|$, so one would have to add additional criteria in order to know what one is talking about. –  Henning Makholm Nov 15 '11 at 20:44
@Henning Good point. Just so it's here, minimizing the $L^1$ norm (i.e. the quantity Henning is talking about) also is in some sense more robust to outliers than the least-squares fit. Provided everything pushes through okay, this is can be quite desirable. –  guy Nov 15 '11 at 20:59
@guy: I'm not quite sure why you consider the $L_1$ minimization "more difficult" to do. It's just a linear program. You won't obtain an analytical solution, in general, but from a numerics perspective, there is not too much lost. –  cardinal Nov 16 '11 at 1:36
@cardinal Yes, I'm aware that it isn't in fact "difficult" computationally - IIRC you can do things like a full least angle regression on the same order of time as it takes to fit a simple linear regression, so these convex problems aren't a big deal these days. I meant analytically you don't get a nice answer, which IMO is the reason it isn't taught. –  guy Nov 16 '11 at 1:55
I agree. It seemed by "difficulty of the fit" you were referring to the actual computation needed. I think the main reason least squares has such a firm place in the classroom is that there is such incredibly rich theory associated with it. –  cardinal Nov 16 '11 at 3:19

Well, you asked for a "simpler" description for the regression in the context of least-squares-fit. I find this a simpler one, although the basic ingredience might be more difficult than the usual approach: thinking in terms of n dimensions, where n is the number of observations, i.e. questioned people or in your case n=5 by the given five points.

If this can be imagined without too much hassle, then the following should also be a "simpler" explanation.

The variables are then vectors from the origin into that n-dimensional space, and we have from your example the points (1,3)(2,8), and extend this to for instance (4,5)(5,7)(3,6) to have n=5 points.
Then we can rewrite this as the vector X pointing to (1,2,4,5,3) in that five dimensions and Y pointing to (3,8,5,7,6).
Also we define the "mean-vector" $\small M(k)$ pointing to any coordinate $\small k \cdot (1,1,1,1,1)$ on the (multidimensional) diagonal.

Clearly $\small M((1+2+4+5+3)/5)=M(3)=3 \cdot (1,1,1,1,1)$ is the vector pointing to the mean of the coordinates of X and $\small M((3+8+5+7+6)/5)=M(29/5)=5.8 \cdot (1,1,1,1,1)$ is the vector pointing to the mean of the coordinates of Y.
Note, that by this the point M(3) is the point on the diagonal which is nearest to the tip of X : but to express the distance of two points we simply need the pythagorean formula with their coordinates - and come up exactly with the least-squares-criterion. This is also true for the distance of M(5.8) from the tip of Y . Just as a spinoff of that model we "see" immediately that the "means" are the best approximates to a set of values in the sense of least-squares.

The same model can now be used to express the linear regression: we want to compose the mean M(.) and the X-vector in such a way that we come nearest to the tip of Y. Or said differently, we want to find a vector $\small \hat Y = a \cdot M(3)+b \cdot X = M(3 a)+b \cdot X$ with a and b to be found such that the tip of $\small \hat Y$ is nearest to the tip of Y.
That's linear regression in the sense of least-squares-approximation.

-