Given $A$ and $\vec{b}$ in $A\vec{x}=\vec{b}$, solve for $\vec{x}$ What are the steps to solve for $\vec{x}$, given that $A\vec{x}=\vec{b}$ and we know what $A$ and $\vec{b}$ are?
I know the first thing you do is multiply each side by $A^T$
$$A^TA\vec{x}=A^T\vec{b}$$
After that I know that if you make an augmented matrix $(A^TA\ |\ A^T\vec{b})$ and put it into RREF the resulting vector is $\vec{x}$. 
But this doesn't make sense to me. What does the transition from $A^TA\vec{x}=A^T\vec{b}$ to $\vec{x}=[somthing]$ look like and why is an augmented matrix the best way to solve this problem? It seems like there is a more formal way to solve for $\vec{x}$.
 A: Think the problem in this terms...
You have $A$ as an an application that transform any $x\in\mathbb{R}^n$ in a vector in $y\in\mathbb{R}^m$. The equation $Ax=b$ is equivalent to the question: "Find the vector $x\in\mathbb{R}^n$ that has $y=Ax$ as image in $\mathbb{R}^m$. Now, there is no guarantee that $b$ is inside the image subspace of $A$...so you can take the vector $x$ that has the nearest to image subspace of transformation $A$. (The transformation $A$ brings whole $\mathbb{R}^n$ in a linear subspace of $\mathbb{R}^m$); in order to find the nearest vector you have to choose a metric, so usually is used the euclidian norm/metric..now from geometry you know that the euclidean metric derives from the standard scalar product and so we can use the orthogonal projection of $b$ on $A$ image subspace. The calculation is very simple..so let's take $y=Ax$, $y\in\mathbb{R}^m$, it (for definition) will be inside the image subspace of $A$. Then we can state that the difference vector $(Ax-b)$ is orthogonal to $Ax$, for orthogonal projection (in order to find the nearest). So we have:
$(Ax-b)^TAx=0,\quad x^TA^TAx-b^TAx=0,\quad x^TA^TAx-x^TA^Tb=0,\quad x^T(A^TAx-A^Tb)=0$
(where scalar is invariant to transposition). So for the independence of $x$ chosen we have finally $A^TAx=A^Tb$. The matrix $A$, if non singular, will bring the solution (euclidian sense): $x=(A^TA)^{-1}A^Tb$
This is the geometrical interpretation, but you can follow the metric idea using derivative (is more analytic and general approach valid also for other metrics).
Forgive for my english and hope it helps.
