# What method does a calculator use to calculate a linear regression line?

Take three coordinates $$(1,1)$$, $$(3,2)$$ and $$(4,3)$$.

My calculator returns the linear regression line: $$y=0.6429x+0.2857$$ of the form $$y = ax +b$$ correct to four significant figures for constants $$a$$ and $$b$$.

How can I do this calculation by hand?

I've heard of least square fitting but I haven't learned how to do that and I'm not sure if it is the method or not.

Can someone point me in the right direction?

Also, please don't suggest I plot the points and draw a best fit line by eye and then get my line from the graph. I want to know what method calculators use to calculate the constants $$a$$ and $$b$$.

• They do this. Apr 20, 2019 at 21:03
• Could you show me with the three coordinates I gave above? Apr 20, 2019 at 21:07
• "Also, please don't suggest I plot the points and draw a best fit line by eye and then get my line from the graph. I want to know what method calculators use to calculate the constants $a$ and $b$." Then don't use tags like geometry, graph-theory and graphing-functions. I can appreciate the sentiment, but you gotta help yourself first. Apr 20, 2019 at 21:08
• Regarding doing this by hand. Are you familiar with the concept of standard deviation? Is this something you know how to find? What about the Pearson correlation coefficient? Apr 20, 2019 at 21:11
• @GitGud Yeah, I am familiar with standard deviation and Pearson's correlation coefficient. Are they used in the calculation of the constants a and b ? Apr 20, 2019 at 21:13

Linear regression is a very general technique, which in this case reduces to

$$\hat{a}=\dfrac{\displaystyle\sum_{i=1}^n(y_i-\bar{y})(x_i-\bar{x})}{\displaystyle\sum_{i=1}^n(x_i-\bar{x})^2},$$

$$\text{and }\hat{b}=\bar{y}-\hat{a}\bar{x},$$

where $$\bar{y}=\dfrac{1}{n}\displaystyle\sum_{i=1}^ny_i,$$ and $$\bar{x}=\dfrac{1}{n}\displaystyle\sum_{i=1}^nx_i.$$

In your case $$y_i$$'s are $$1,2,3$$ and $$x_i$$'s are $$1,3,4$$ for $$i=1,2,3$$ respectively.

• That works perfectly and answers my question fully. Thank you. Apr 20, 2019 at 21:25
• What is the name of that technique ? Apr 20, 2019 at 21:37
• Simple linear regression. Wikipedia Apr 20, 2019 at 21:53

If you want to fit a line to a list of points $$(x_i, y_i), i = 1, \ldots, N$$, the most popular approach is the "least squares" approach, wherein $$a$$ and $$b$$ are chosen to minimize the sum of squared residuals: $$\min_{a,b} \quad \sum_{i=1}^N (a x_i + b - y_i)^2.$$ You can minimize this sum of squared residuals by setting the partial derivatives with respect to $$a$$ and $$b$$ equal to $$0$$, and then solving the resulting linear system of equations for $$a$$ and $$b$$.

(Taken from a previous writeup)

How to do linear least squares fitting.

To fit a linear sum of $$m$$ functions $$f_k(x), k=1$$ to $$m$$ to $$n$$ points $$(x_i, y_i), i=1$$ to $$n$$, we want to find the $$a_k, k=1$$ to $$m$$ so that $$\sum_{k=1}^m a_kf_k(x)$$ best fits the data.

Let $$S =\sum_{i=1}^n(y_i-\sum_{k=1}^m a_kf_k(x_i))^2$$.

$$\begin{array}\\ \dfrac{\partial S}{\partial a_j} &=D_jS\\ &=D_j\sum_{i=1}^n(y_i-\sum_{k=1}^m a_kf_k(x_i))^2\\ &=\sum_{i=1}^nD_j(y_i-\sum_{k=1}^m a_kf_k(x_i))^2\\ &=\sum_{i=1}^n2(y_i-\sum_{k=1}^m a_kf_k(x_i))D_j(y_i-\sum_{k=1}^m a_kf_k(x_i))\\ &=\sum_{i=1}^n2(y_i-\sum_{k=1}^m a_kf_k(x_i))(-D_j a_jf_j(x_i))\\ &=\sum_{i=1}^n2(y_i-\sum_{k=1}^m a_kf_k(x_i))(- f_j(x_i))\\ &=-2\sum_{i=1}^nf_j(x_i)(y_i-\sum_{k=1}^m a_kf_k(x_i))\\ &=-2\left(\sum_{i=1}^ny_if_j(x_i)-\sum_{i=1}^nf_j(x_i)\sum_{k=1}^m a_kf_k(x_i)\right)\\ &=-2\left(\sum_{i=1}^ny_if_j(x_i)-\sum_{k=1}^m a_k\sum_{i=1}^nf_j(x_i)f_k(x_i)\right)\\ \end{array}$$

Therefore, if $$D_jS = 0$$, then $$\sum_{i=1}^ny_if_j(x_i) =\sum_{k=1}^m a_k\sum_{i=1}^nf_j(x_i)f_k(x_i)$$.

Doing this for $$j=1$$ to $$m$$ gives $$m$$ equations in the $$m$$ unknowns $$a_1, ..., a_m$$.

Example: To fit a polynomial of degree $$m-1$$, let $$f_j(x) = x^{j-1}$$. The equations are then

$$\begin{array}\\ \sum_{i=1}^ny_ix_i^{j-1} &=\sum_{k=1}^m a_k\sum_{i=1}^nx_i^{j-1}x_i^{k-1}\\ &=\sum_{k=1}^m a_k\sum_{i=1}^nx_i^{k+j-2}\\ \end{array}$$

For a line, $$m=2$$ and the equations are, for $$j = 1, 2$$,

$$\begin{array}\\ \sum_{i=1}^ny_ix_i^{j-1} &=\sum_{k=1}^2 a_k\sum_{i=1}^nx_i^{k+j-2}\\ &= a_1\sum_{i=1}^nx_i^{j-1}+a_2\sum_{i=1}^nx_i^{j}\\ \end{array}$$

Explicitly these are

$$j=1:\sum_{i=1}^ny_i = a_1n+a_2\sum_{i=1}^nx_i\\ j=2:\sum_{i=1}^nx_iy_i = a_1\sum_{i=1}^nx_i+a_2\sum_{i=1}^nx_i^{2}\\$$

These should look familiar.

For a quadratic, $$m=3$$ and the equations are, for $$j = 1, 2, 3$$,

$$\begin{array}\\ \sum_{i=1}^ny_ix_i^{j-1} &=\sum_{k=1}^3 a_k\sum_{i=1}^nx_i^{k+j-2}\\ &= a_1\sum_{i=1}^nx_i^{j-1}+a_2\sum_{i=1}^nx_i^{j}+a_3\sum_{i=1}^nx_i^{j+1}\\ \end{array}$$

Example 2. To fit a line through the origin, $$y = ax$$, $$m=1$$ and $$f_1(x) = x$$. The equation is then $$\sum_{i=1}^ny_ix_i =a_1\sum_{i=1}^nx_i^2$$ so the result is $$a =\dfrac{\sum_{i=1}^nx_iy_i}{\sum_{i=1}^nx_i^2}$$.

• The calculation is much nicer if we use matrix and vector notation. We can write $S = \| y - F a \|^2$, and then minimize $S$ by setting the gradient equal to $0$: $\nabla S(a) = -2 F^T(y-Fa) = 0 \implies F^T Fa = F^Ty$. This linear system of equations can be solved using a method such as Gaussian elimination. Apr 21, 2019 at 8:52
• That is true, and I upvoted your comment. However, for those not familiar with matrices and vectors, I think the algebraic way I have done is easier to understand. Apr 21, 2019 at 20:16