Solution to Ax=b with Least Squares Let's suppose $A$ is a matrix such that $\ker A=\{0\}$ and $b$ is a vector not in the image of $A$.  This situation implies $Ax=b$ does not have a solution.
This is where least squares comes in.  Since $b$ is not in the image, we project $b$ onto the image of $A$ to get some vector $b'$ where then the equation $Ax=b'$ is solvable by $\hat{x}$.  This $\hat{x}$ is not the least-squares solution to $Ax=b$.
My question is, if the columns of $A$ are linearly independent and we project $b$ to get $b'\in image(A)$, then why can't we just row-reduce $Ax=b'$ to get the solution $\hat{x}?$  Instead, we must solve the normal equation $A^TAx=A^Tb.$  I get that $(A^TA)^{-1}A^T$ is a projection map, but I am looking for something more fundamental.
 A: Remember : Since you estimating a solution using LSQ, the more the data you account for, the better.
Example.
Suppose you have 3 variables.
Theoretically, you need only $3$ equations i.e. a $3\times 3 $ matrix is what you need. 
However if you have one more equation from the data collected i.e. a $4\times 3$ matrix, theoritically the last equation should be a linear combination of the above $3$ . However practiacally, (quite obvious) things will differ.
So, now since the columns are independent, if you go by row-reduction , you will take account of only the first $3$ equations, i.e. the data from the $4th$ equation is not taken into account.
However , If you multiply by $A^t$, you get a square matrix, and it will be invertible (WHY?) .But now, you have accounted for the entire data!
So the latter ( by Remember ) will give a better approxiamtion than Row-reduction.
A: You could. The result would be the same. Solving the normal equations is just one method.
A: Two things:


*

*The equation $Ax = b'$ doesn't always make sense, for dimensionality reasons. If $A$ is not square, then $b$ and $b'$ will be different lengths. 

*Even if $A$ is square and thus the equation $Ax = b'$ does make sense, solving $A^tAx = b'$ still gives a more optimal solution. Proving why it is optimal is complicated.
A: The least-square approach is based on the minimization of the quadratic error,
$$E=\|Ax-b\|^2=(Ax-b)^T(Ax-b).$$
Differentiating on $x$, we cancel
$$\dfrac{dE}{dx}=2A^T(Ax-b)$$
so that
$$A^TAx=A^Tb.$$
The error minimization is achieved by an orthogonal projection.
