1
$\begingroup$

Take three coordinates $(1,1)$, $(3,2)$ and $(4,3)$.

My calculator returns the linear regression line: $$y=0.6429x+0.2857$$ of the form $$y = ax +b$$ correct to four significant figures for constants $a$ and $b$.

How can I do this calculation by hand?

I've heard of least square fitting but I haven't learned how to do that and I'm not sure if it is the method or not.

Can someone point me in the right direction?

Also, please don't suggest I plot the points and draw a best fit line by eye and then get my line from the graph. I want to know what method calculators use to calculate the constants $a$ and $b$.

$\endgroup$
6
  • $\begingroup$ They do this. $\endgroup$
    – user647486
    Apr 20, 2019 at 21:03
  • $\begingroup$ Could you show me with the three coordinates I gave above? $\endgroup$
    – Kantura
    Apr 20, 2019 at 21:07
  • $\begingroup$ "Also, please don't suggest I plot the points and draw a best fit line by eye and then get my line from the graph. I want to know what method calculators use to calculate the constants $a$ and $b$." Then don't use tags like geometry, graph-theory and graphing-functions. I can appreciate the sentiment, but you gotta help yourself first. $\endgroup$
    – Git Gud
    Apr 20, 2019 at 21:08
  • $\begingroup$ Regarding doing this by hand. Are you familiar with the concept of standard deviation? Is this something you know how to find? What about the Pearson correlation coefficient? $\endgroup$
    – Git Gud
    Apr 20, 2019 at 21:11
  • $\begingroup$ @GitGud Yeah, I am familiar with standard deviation and Pearson's correlation coefficient. Are they used in the calculation of the constants a and b ? $\endgroup$
    – Kantura
    Apr 20, 2019 at 21:13

3 Answers 3

1
$\begingroup$

Linear regression is a very general technique, which in this case reduces to

$$\hat{a}=\dfrac{\displaystyle\sum_{i=1}^n(y_i-\bar{y})(x_i-\bar{x})}{\displaystyle\sum_{i=1}^n(x_i-\bar{x})^2},$$

$$\text{and }\hat{b}=\bar{y}-\hat{a}\bar{x},$$

where $\bar{y}=\dfrac{1}{n}\displaystyle\sum_{i=1}^ny_i,$ and $\bar{x}=\dfrac{1}{n}\displaystyle\sum_{i=1}^nx_i.$

In your case $y_i$'s are $1,2,3$ and $x_i$'s are $1,3,4$ for $i=1,2,3$ respectively.

$\endgroup$
3
  • $\begingroup$ That works perfectly and answers my question fully. Thank you. $\endgroup$
    – Kantura
    Apr 20, 2019 at 21:25
  • $\begingroup$ What is the name of that technique ? $\endgroup$
    – Kantura
    Apr 20, 2019 at 21:37
  • $\begingroup$ Simple linear regression. Wikipedia $\endgroup$ Apr 20, 2019 at 21:53
1
$\begingroup$

If you want to fit a line to a list of points $(x_i, y_i), i = 1, \ldots, N$, the most popular approach is the "least squares" approach, wherein $a$ and $b$ are chosen to minimize the sum of squared residuals: $$ \min_{a,b} \quad \sum_{i=1}^N (a x_i + b - y_i)^2. $$ You can minimize this sum of squared residuals by setting the partial derivatives with respect to $a$ and $b$ equal to $0$, and then solving the resulting linear system of equations for $a$ and $b$.

$\endgroup$
0
$\begingroup$

(Taken from a previous writeup)

How to do linear least squares fitting.

To fit a linear sum of $m$ functions $f_k(x), k=1$ to $m$ to $n$ points $(x_i, y_i), i=1$ to $n$, we want to find the $a_k, k=1$ to $m$ so that $\sum_{k=1}^m a_kf_k(x) $ best fits the data.

Let $S =\sum_{i=1}^n(y_i-\sum_{k=1}^m a_kf_k(x_i))^2$.

$\begin{array}\\ \dfrac{\partial S}{\partial a_j} &=D_jS\\ &=D_j\sum_{i=1}^n(y_i-\sum_{k=1}^m a_kf_k(x_i))^2\\ &=\sum_{i=1}^nD_j(y_i-\sum_{k=1}^m a_kf_k(x_i))^2\\ &=\sum_{i=1}^n2(y_i-\sum_{k=1}^m a_kf_k(x_i))D_j(y_i-\sum_{k=1}^m a_kf_k(x_i))\\ &=\sum_{i=1}^n2(y_i-\sum_{k=1}^m a_kf_k(x_i))(-D_j a_jf_j(x_i))\\ &=\sum_{i=1}^n2(y_i-\sum_{k=1}^m a_kf_k(x_i))(- f_j(x_i))\\ &=-2\sum_{i=1}^nf_j(x_i)(y_i-\sum_{k=1}^m a_kf_k(x_i))\\ &=-2\left(\sum_{i=1}^ny_if_j(x_i)-\sum_{i=1}^nf_j(x_i)\sum_{k=1}^m a_kf_k(x_i)\right)\\ &=-2\left(\sum_{i=1}^ny_if_j(x_i)-\sum_{k=1}^m a_k\sum_{i=1}^nf_j(x_i)f_k(x_i)\right)\\ \end{array} $

Therefore, if $D_jS = 0$, then $\sum_{i=1}^ny_if_j(x_i) =\sum_{k=1}^m a_k\sum_{i=1}^nf_j(x_i)f_k(x_i) $.

Doing this for $j=1$ to $m$ gives $m$ equations in the $m$ unknowns $a_1, ..., a_m$.

Example: To fit a polynomial of degree $m-1$, let $f_j(x) = x^{j-1}$. The equations are then

$\begin{array}\\ \sum_{i=1}^ny_ix_i^{j-1} &=\sum_{k=1}^m a_k\sum_{i=1}^nx_i^{j-1}x_i^{k-1}\\ &=\sum_{k=1}^m a_k\sum_{i=1}^nx_i^{k+j-2}\\ \end{array} $

For a line, $m=2$ and the equations are, for $j = 1, 2$,

$\begin{array}\\ \sum_{i=1}^ny_ix_i^{j-1} &=\sum_{k=1}^2 a_k\sum_{i=1}^nx_i^{k+j-2}\\ &= a_1\sum_{i=1}^nx_i^{j-1}+a_2\sum_{i=1}^nx_i^{j}\\ \end{array} $

Explicitly these are

$j=1:\sum_{i=1}^ny_i = a_1n+a_2\sum_{i=1}^nx_i\\ j=2:\sum_{i=1}^nx_iy_i = a_1\sum_{i=1}^nx_i+a_2\sum_{i=1}^nx_i^{2}\\ $

These should look familiar.

For a quadratic, $m=3$ and the equations are, for $j = 1, 2, 3$,

$\begin{array}\\ \sum_{i=1}^ny_ix_i^{j-1} &=\sum_{k=1}^3 a_k\sum_{i=1}^nx_i^{k+j-2}\\ &= a_1\sum_{i=1}^nx_i^{j-1}+a_2\sum_{i=1}^nx_i^{j}+a_3\sum_{i=1}^nx_i^{j+1}\\ \end{array} $

Example 2. To fit a line through the origin, $y = ax$, $m=1$ and $f_1(x) = x$. The equation is then $\sum_{i=1}^ny_ix_i =a_1\sum_{i=1}^nx_i^2 $ so the result is $a =\dfrac{\sum_{i=1}^nx_iy_i}{\sum_{i=1}^nx_i^2} $.

$\endgroup$
2
  • 1
    $\begingroup$ The calculation is much nicer if we use matrix and vector notation. We can write $S = \| y - F a \|^2$, and then minimize $S$ by setting the gradient equal to $0$: $\nabla S(a) = -2 F^T(y-Fa) = 0 \implies F^T Fa = F^Ty$. This linear system of equations can be solved using a method such as Gaussian elimination. $\endgroup$
    – littleO
    Apr 21, 2019 at 8:52
  • 1
    $\begingroup$ That is true, and I upvoted your comment. However, for those not familiar with matrices and vectors, I think the algebraic way I have done is easier to understand. $\endgroup$ Apr 21, 2019 at 20:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .