What method does a calculator use to calculate a linear regression line? Take three coordinates $(1,1)$, $(3,2)$ and $(4,3)$.
My calculator returns the linear regression line: $$y=0.6429x+0.2857$$ of the form $$y = ax +b$$ correct to four significant figures for constants $a$ and $b$.
How can I do this calculation by hand?
I've heard of least square fitting but I haven't learned how to do that and I'm not sure if it is the method or not.
Can someone point me in the right direction?
Also, please don't suggest I plot the points and draw a best fit line by eye and then get my line from the graph. I want to know what method calculators use to calculate the constants $a$ and $b$.
 A: Linear regression is a very general technique, which in this case reduces to
$$\hat{a}=\dfrac{\displaystyle\sum_{i=1}^n(y_i-\bar{y})(x_i-\bar{x})}{\displaystyle\sum_{i=1}^n(x_i-\bar{x})^2},$$
$$\text{and }\hat{b}=\bar{y}-\hat{a}\bar{x},$$
where $\bar{y}=\dfrac{1}{n}\displaystyle\sum_{i=1}^ny_i,$ and $\bar{x}=\dfrac{1}{n}\displaystyle\sum_{i=1}^nx_i.$
In your case $y_i$'s are $1,2,3$ and $x_i$'s are $1,3,4$ for $i=1,2,3$ respectively.
A: If you want to fit a line to a list of points $(x_i, y_i), i = 1, \ldots, N$, the most popular approach is the "least squares" approach, wherein $a$ and $b$ are chosen to minimize the sum of squared residuals:
$$
\min_{a,b} \quad \sum_{i=1}^N (a x_i + b - y_i)^2.
$$
You can minimize this sum of squared residuals by setting the partial derivatives with respect to $a$ and $b$ equal to $0$, and then solving the resulting linear system of equations for $a$ and $b$.
A: (Taken from a previous writeup)
How to do
linear least squares fitting.
To fit
a linear sum of
$m$ functions
$f_k(x), k=1$ to $m$
to $n$ points
$(x_i, y_i), i=1$ to $n$,
we want to find the
$a_k, k=1$ to $m$
so that
$\sum_{k=1}^m a_kf_k(x)
$
best fits the data.
Let
$S
=\sum_{i=1}^n(y_i-\sum_{k=1}^m a_kf_k(x_i))^2$.
$\begin{array}\\
\dfrac{\partial S}{\partial a_j}
&=D_jS\\
&=D_j\sum_{i=1}^n(y_i-\sum_{k=1}^m a_kf_k(x_i))^2\\
&=\sum_{i=1}^nD_j(y_i-\sum_{k=1}^m a_kf_k(x_i))^2\\
&=\sum_{i=1}^n2(y_i-\sum_{k=1}^m a_kf_k(x_i))D_j(y_i-\sum_{k=1}^m a_kf_k(x_i))\\
&=\sum_{i=1}^n2(y_i-\sum_{k=1}^m a_kf_k(x_i))(-D_j a_jf_j(x_i))\\
&=\sum_{i=1}^n2(y_i-\sum_{k=1}^m a_kf_k(x_i))(- f_j(x_i))\\
&=-2\sum_{i=1}^nf_j(x_i)(y_i-\sum_{k=1}^m a_kf_k(x_i))\\
&=-2\left(\sum_{i=1}^ny_if_j(x_i)-\sum_{i=1}^nf_j(x_i)\sum_{k=1}^m a_kf_k(x_i)\right)\\
&=-2\left(\sum_{i=1}^ny_if_j(x_i)-\sum_{k=1}^m a_k\sum_{i=1}^nf_j(x_i)f_k(x_i)\right)\\
\end{array}
$
Therefore,
if $D_jS = 0$,
then
$\sum_{i=1}^ny_if_j(x_i)
=\sum_{k=1}^m a_k\sum_{i=1}^nf_j(x_i)f_k(x_i)
$.
Doing this
for $j=1$ to $m$
gives $m$ equations
in the $m$ unknowns
$a_1, ..., a_m$.
Example:
To fit a polynomial
of degree $m-1$,
let $f_j(x) = x^{j-1}$.
The equations are then
$\begin{array}\\
\sum_{i=1}^ny_ix_i^{j-1}
&=\sum_{k=1}^m a_k\sum_{i=1}^nx_i^{j-1}x_i^{k-1}\\
&=\sum_{k=1}^m a_k\sum_{i=1}^nx_i^{k+j-2}\\
\end{array}
$
For a line,
$m=2$
and the equations are,
for
$j = 1, 2$,
$\begin{array}\\
\sum_{i=1}^ny_ix_i^{j-1}
&=\sum_{k=1}^2 a_k\sum_{i=1}^nx_i^{k+j-2}\\
&= a_1\sum_{i=1}^nx_i^{j-1}+a_2\sum_{i=1}^nx_i^{j}\\
\end{array}
$
Explicitly these are
$j=1:\sum_{i=1}^ny_i
= a_1n+a_2\sum_{i=1}^nx_i\\
j=2:\sum_{i=1}^nx_iy_i
= a_1\sum_{i=1}^nx_i+a_2\sum_{i=1}^nx_i^{2}\\
$
These should look familiar.
For a quadratic,
$m=3$
and the equations are,
for
$j = 1, 2, 3$,
$\begin{array}\\
\sum_{i=1}^ny_ix_i^{j-1}
&=\sum_{k=1}^3 a_k\sum_{i=1}^nx_i^{k+j-2}\\
&= a_1\sum_{i=1}^nx_i^{j-1}+a_2\sum_{i=1}^nx_i^{j}+a_3\sum_{i=1}^nx_i^{j+1}\\
\end{array}
$
Example 2.
To fit a line
through the origin,
$y = ax$,
$m=1$ and
$f_1(x) = x$.
The equation is then
$\sum_{i=1}^ny_ix_i
=a_1\sum_{i=1}^nx_i^2
$
so the result is
$a
=\dfrac{\sum_{i=1}^nx_iy_i}{\sum_{i=1}^nx_i^2}
$.
