Let $A\in \mathbb{R^{m\times n}}$ have full column rank, $b\in \mathbb{R^m}$ with $b \neq 0$

Show that $P = A(A^TA)^{-1}A^T$ is a projection matrix. Onto what space does $P$ project?

I tried looking at videos,at textbooks, and lecture notes. Which version best answers the question?

version 1:

Since $ATA$ is an invertible matrix, take the normal equation $A^TAx=A^Tb$ and solve for $x$. We get, $x=(A^TA)^{-1}A^Tb$. For $Ax=Pb$, we have $Ax$ as a projection, and $P$ is an orthogonal projector onto the range of $A$. It follows, $A(A^TA)^{-1}A^Tb = Pb$, $\forall b$. Generally, for any matrices $M$ and $N$ s.t. $Mx=Nx$,$\forall x \Rightarrow M=N$, thus $P=A(A^TA)^{-1}A^T$. $P$ projects a vector $b$ onto the spaced spanned by the columns of $A$.

version 2

Given a matrix $A$, the projection of $b$ onto $C(A)$ is $P=Ax$, where $x$ solves $A^TAx=A^Tb$. Since A has full column rank, $A^TA$ is invertible. We write, the projection matrix is $P=A(A^TA)^{-1}A^T$.

version 3

The vector $Ax$ is always in the column space of $A$. We project b onto a vector $P$ in the column space of $A$ and solve $Ax=P$. In matrix form, $A^T(b-Ax)=0 \Rightarrow A^TAx=A^Tb$. Since $A^TA$ is a square matrix, we multiply by $(A^TA)^{-1}$ In $n$ dimensions, $x=(A^TA)^{-1}A^Tb \Rightarrow P=Ax=A(A^TA)^{-1}A^Tb \Rightarrow P=A(A^TA)^{-1}A^T$

  • 2
    $\begingroup$ The simplest way to show that $P$ is a projection is to use the most basic definition of one: $P^2=P$. For the second part, consider the product $PA$. $\endgroup$ – amd Mar 9 '17 at 3:18
  • 1
    $\begingroup$ In fact, $P = A(A^T A)^{-1}A^T$ is called the projector matrix onto the column space of $A$. $\endgroup$ – mathJuan Mar 9 '17 at 3:25
  • $\begingroup$ @amd indeed, the question is to explain $P = A(A^TA)^{-1}A^T$ perhaps using the fact that A is full column rank or $A^TA$ is invertible. $\endgroup$ – David Mar 9 '17 at 3:38
  • 1
    $\begingroup$ @DivVares Deriving this expression for $P$ is an interesting exercise, but not the question at hand. $\endgroup$ – amd Mar 9 '17 at 3:41
  • 2
    $\begingroup$ @DivVares Good point, though. For $A^TA$ to be nonsingular, $A$ must have full column rank, not simply full rank. $\endgroup$ – amd Mar 9 '17 at 3:44

Throughout versions 2 and 3, you confuse the projected vector $p = Pb$ with the matrix $P$; you should avoid using the same symbol to mean both. I ignore that point in the following.

I'd say that the most appropriate version depends on the context. Version 1 is good in that it emphasizes the normal equation, which may be the focus of a few weeks of a class in and of itself. Version 2 is particularly terse, which can be desirable; since you are still studying the material, I would avoid this one. Version 3 exploits the fact that $\mathcal C(A)^\perp = \mathcal N(A^T)$, but fails to make this explicit. You also have some unclear word choice.

All together, I would say that, as written, your first version is the most cohesive and complete. I personally prefer the approach you take in the third, but your current writeup requires a lot of clarification.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.