I have always been confused about whether the approximate solution to $Ax=b$ is equivalent to minimizing the average distance of all of the $b$ vectors to $Ax$, or whether it is minimizing the distance projected along the $b$ axis?
(where $A$ is full rank and skinny and the system is overdetermined).
Consider these two pictures:
from http://www.statisticshowto.com/least-squares-regression-line/
and another figure on the same page:
also from from http://www.statisticshowto.com/least-squares-regression-line/.
Notice these are two figures on the same page! I do not understand how these are both being minimized at the same time. can someone please explain? thanks.
this is a related question Least squares solutions and orthogonal projection?