Using least squares regression for line of best fit Use the least square approximation to find the closest line (the line of "Best Fit") to the points:
$$(-6,-1), \quad (-2,2), \quad (1,1), \quad (7,6)$$
I'm attempting to use the least squares approximation formulation that is as follows:
$$A^TAx = A^Tb$$
However, I'm confused because I'm given four vectors. Does that mean I can use the first three vectors $(-6,-1),(-2,2),(1,1)$ to create my matrix A, and then use the last vector $(7,6)$ for the "$b$" value? I understand the process is very straight forward once proper substitution has been done, but I tried the method I described above and got the wrong answer.Thank you guys. 
 A: You are looking for an equation $y=mx+c$. Ideally it would pass through all of the given points: that is, you would have $-6m+c = -1$, and similarly for other points. This is a set of four equations with two unknowns $m,c$. Its matrix representation is
$$
\begin{pmatrix}
-6 & 1\\ -2 & 1 \\ 1 & 1 \\ 7 & 1
\end{pmatrix}
\begin{pmatrix}
m \\ c
\end{pmatrix} 
 = \begin{pmatrix}
-1 \\ 2 \\ 1 \\6
\end{pmatrix} 
$$ 
These are $A$ and $b$ you are looking for. Typically, the system $Ax=b$ has no solutions (there is no line through all the points); this is why  we solve $A^TAx=A^Tb$ instead, obtaining $x$ that minimizes the residual $\|Ax-b\|^2$.
A: The trial function is
$$
  y(x) = c_{0} + c_{1} x.
$$
As noted by @user147263,you have the linear system
$$
\begin{align}
  \mathbf{A} c & = y\\
%
\left[
\begin{array}{rr}
 1 & -6 \\
 1 & -2 \\
 1 & 1 \\
 1 & 7 \\
\end{array}
\right]
%
\left[
\begin{array}{r}
 c_{0} \\
 c_{1} \\
\end{array}
\right] 
%
  &=
%
\left[
\begin{array}{r}
 -1 \\
  2 \\
  1 \\
  6
\end{array}
\right].
%
\end{align}
$$
Your choice for solution is the normal equations
$$
\begin{align}
%
  \mathbf{A}^{*} \mathbf{A} c &= \mathbf{A}^{*}y \\
%
\left[
\begin{array}{cc}
 4 & 0 \\
 0 & 90 
\end{array}
\right]
%
%
\left[
\begin{array}{r}
 c_{0} \\
 c_{1} \\
\end{array}
\right] 
%
  &=
%
\left[
\begin{array}{r}
  8 \\
 45 
\end{array}
\right] .
%
\end{align}
$$
The solution is 
$$
\begin{align}
  c &= \left( \mathbf{A}^{*} \mathbf{A} \right)^{-1} \mathbf{A}^{*} y \\
%
\left[
\begin{array}{r}
 c_{0} \\
 c_{1} \\
\end{array}
\right] 
%
  &=
%
\left[
\begin{array}{cc}
 \frac{1}{4} & 0 \\
 0 & \frac{1}{90} \\
\end{array}
\right]
%
\left[
\begin{array}{r}
  8 \\
 45 
\end{array}
\right] \\
%
 &=
%
\left[
\begin{array}{c}
  2 \\
 \frac{1}{2} 
\end{array}
\right] .
%
\end{align}
$$
The solution function is
$$
 y(x) = 2 + \frac{1}{2} x.
$$
The residual error vector is 
$$
  r = \mathbf{A}c - 7 =
\left[
\begin{array}{r}
  0 \\
 -1 \\
  \frac{3}{2} \\
 -\frac{1}{2} 
\end{array}
\right]
$$
with a total error of $r^{2} = \frac{7}{2}.$
The solution is plotted against the data points below.

