# Show that $P = A(A^TA)^{-1}A^T$ is a projection matrix.

Let $A\in \mathbb{R^{m\times n}}$ have full column rank, $b\in \mathbb{R^m}$ with $b \neq 0$

Show that $P = A(A^TA)^{-1}A^T$ is a projection matrix. Onto what space does $P$ project?

I tried looking at videos,at textbooks, and lecture notes. Which version best answers the question?

version 1:

Since $ATA$ is an invertible matrix, take the normal equation $A^TAx=A^Tb$ and solve for $x$. We get, $x=(A^TA)^{-1}A^Tb$. For $Ax=Pb$, we have $Ax$ as a projection, and $P$ is an orthogonal projector onto the range of $A$. It follows, $A(A^TA)^{-1}A^Tb = Pb$, $\forall b$. Generally, for any matrices $M$ and $N$ s.t. $Mx=Nx$,$\forall x \Rightarrow M=N$, thus $P=A(A^TA)^{-1}A^T$. $P$ projects a vector $b$ onto the spaced spanned by the columns of $A$.

version 2

Given a matrix $A$, the projection of $b$ onto $C(A)$ is $P=Ax$, where $x$ solves $A^TAx=A^Tb$. Since A has full column rank, $A^TA$ is invertible. We write, the projection matrix is $P=A(A^TA)^{-1}A^T$.

version 3

The vector $Ax$ is always in the column space of $A$. We project b onto a vector $P$ in the column space of $A$ and solve $Ax=P$. In matrix form, $A^T(b-Ax)=0 \Rightarrow A^TAx=A^Tb$. Since $A^TA$ is a square matrix, we multiply by $(A^TA)^{-1}$ In $n$ dimensions, $x=(A^TA)^{-1}A^Tb \Rightarrow P=Ax=A(A^TA)^{-1}A^Tb \Rightarrow P=A(A^TA)^{-1}A^T$

• The simplest way to show that $P$ is a projection is to use the most basic definition of one: $P^2=P$. For the second part, consider the product $PA$. – amd Mar 9 '17 at 3:18
• In fact, $P = A(A^T A)^{-1}A^T$ is called the projector matrix onto the column space of $A$. – mathJuan Mar 9 '17 at 3:25
• @amd indeed, the question is to explain $P = A(A^TA)^{-1}A^T$ perhaps using the fact that A is full column rank or $A^TA$ is invertible. – David Mar 9 '17 at 3:38
• @DivVares Deriving this expression for $P$ is an interesting exercise, but not the question at hand. – amd Mar 9 '17 at 3:41
• @DivVares Good point, though. For $A^TA$ to be nonsingular, $A$ must have full column rank, not simply full rank. – amd Mar 9 '17 at 3:44

## 1 Answer

Throughout versions 2 and 3, you confuse the projected vector $p = Pb$ with the matrix $P$; you should avoid using the same symbol to mean both. I ignore that point in the following.

I'd say that the most appropriate version depends on the context. Version 1 is good in that it emphasizes the normal equation, which may be the focus of a few weeks of a class in and of itself. Version 2 is particularly terse, which can be desirable; since you are still studying the material, I would avoid this one. Version 3 exploits the fact that $\mathcal C(A)^\perp = \mathcal N(A^T)$, but fails to make this explicit. You also have some unclear word choice.

All together, I would say that, as written, your first version is the most cohesive and complete. I personally prefer the approach you take in the third, but your current writeup requires a lot of clarification.