Proof that the Matrix of Orthogonal Projection can be calculated by $P = A (A^t A)^{-1} A^t$. Prove that the Matrix of Orthogonal Projection $P$ in any subspace $W$, can be calculated by $P = A (A^t A)^{-1} A^t$ where ColA forms a basis for W.
Am absolutely lost in this one, please help.
 A: Okay. Let's take it one step at a time then.
First, let's prove that $P$ is a projection, that is to say $P=P^2$. Then, that $P$ is an orthogonal projection, i.e. that $P^t=P$. Lastly, let's prove that $P$ has the correct image.
So
\begin{align}
P^2 &=(A(A^tA)^{-1}A^t)^2=A(A^tA)^{-1}A^tA(A^tA)^{-1}A^t=A(A^tA)^{-1}A^t \\
P^t &= (A(A^tA)^{-1}A^t)^t=(A^t)^t((A^tA)^{-1})^tA^t=A((A^tA)^t)^{-1}A^t=A(A^tA)^{-1}A^t
\end{align}
Finally, we need to argue for the image of $P$. For this, you only need the fact that $A^t$ is surjective (which you must know in order to know that $A^tA$ is invertible).
A: Essentially, at the heart of the result is that $B = (A^\top A)^{-1} A^\top$ is the Moore-Penrose inverse of $A$ whenever $(A^\top A)^{-1}$ exists (which is when the columns of $A$ are linearly independent). It's OK if you don't know this term, because it's defined by $4$ simple properties:

*

*$ABA = A$

*$BAB = B$

*$(AB)^* = AB$

*$(BA)^* = BA$
As it turns out, for every matrix $A$, there is exactly one matrix $B$ satisfying all four conditions above, and that matrix is the Moore-Penrose inverse of $A$. If $A$ is invertible, then $B = A^{-1}$. We don't need to prove any of this, but if we have a $B$ that satisfies the above four conditions (which we do: check them using our formula for $B$!), then $P = AB$ is the projection matrix onto the columnspace of $A$. So, we should be able to prove this using only the above four properties.
First, note that $P$ is a projection matrix, as
$$P^2 = A\color{red}{BAB} = A\color{red}B = P,$$
using the second property. Next, $P$ is self-adjoint, which is precisely the third property, making $P$ an orthogonal projection onto its columnspace. The only thing we now need to prove is that $\operatorname{colspace}(P) = \operatorname{colspace}(A)$.
Recall that one way to represent the columnspace of a matrix is
$$\operatorname{colspace}(M) = \{Mx : x \text{ is a column vector of appropriate size}\}.$$
So, the columnspace of $P$ is made up of vectors of the form $Px = A(Bx)$, which are elements of $\operatorname{colspace}(A)$ by the above characterisation.
On the other hand, if we take arbitrary $Ax \in \operatorname{colspace}(A)$, then we can write
$$Ax = ABAx = P(Ax) \in \operatorname{colspace}(P),$$
using the first property. So, our column spaces for $A$ and $P$ coincide, hence $P$ is the orthogonal projection matrix onto $\operatorname{colspace}(A)$.
