# Do all projections matrices take this form?

Do all projection matrices take the form $$P = A{(A^TA)}^{-1}A^T$$? If so, can you help me derive it and explain it intuitively?

$$A(A^TA)^{-1}A^T$$ is symmetric, but not all projection matrices are symmetric --- such as $$\pmatrix{1&1\\ 0&0}$$. Thus the answer is clearly no.

It is true, however, that all orthogonal (with respect to the usual inner product) projection matrices over $$\mathbb R$$ can be written in the form of $$A(A^TA)^{-1}A^T$$. By definition, if $$P\in M_n(\mathbb R)$$ is an orthogonal projection, then $$P|_U=\operatorname{id}$$ and $$P|_{U^\perp}=0$$ for some subspace $$U\subseteq\mathbb R^n$$. Let $$A$$ be any matrix whose columns form a basis of $$U$$ (any basis will do; it doesn't have to be orthonormal). Then $$A^TA$$ is nonsingular and $$A(A^TA)^{-1}A^Tv=0$$ for every $$v\in U^\perp$$. Also, since the columns of $$A$$ span $$U$$, every vector $$u\in U$$ can be written as $$Ax$$ for some $$x\in\mathbb R^n$$. Therefore $$\left(A(A^TA)^{-1}A^T\right)u=\left(A(A^TA)^{-1}A^T\right)(Ax)=\left(A(A^TA)^{-1}A^TA\right)x=Ax=u$$ for every $$u=Ax\in U$$. Hence $$P$$ and $$A(A^TA)^{-1}A^T$$ agree everywhere on $$\mathbb R^n$$, i.e. $$P=A(A^TA)^{-1}A^T$$.

• Thanks, I learned something from your answer -- I had forgotten that a projection matrix does not have to be an orthogonal projection matrix. – littleO Jan 1 at 15:24
• You appear to be assuming the standard basis and Euclidean scalar product. In a different basis of the ambient space or with a different scalar product, the expression still yields the matrix of a projection ($P^2=P$ always), but it’s not likely to be orthogonal. – amd Jan 2 at 7:05
• @amd I am. This is the usual setting when this kind of projections are encountered. – user1551 Jan 2 at 9:36

As pointed out by @user1551, this is only true for orthogonal projection matrices.

Let $$P$$ be the orthogonal projection operator that projects a vector $$b \in \mathbb R^n$$ onto a subspace $$S \subset \mathbb R^n$$. Let $$(a_1,\ldots,a_m)$$ be a basis for $$S$$, and let $$A$$ be the matrix whose $$i$$th column is $$a_i$$. Then $$S =\{Ax \mid x \in \mathbb R^m\}$$, and projecting $$b$$ onto $$S$$ is equivalent to selecting $$x$$ so as to minimize the distance from $$b$$ to $$Ax$$. Equivalently, we want to minimize $$r(x) = \| Ax - b \|^2.$$ This is a least squares problem. Setting the gradient equal to $$0$$, we find that $$x$$ satisfies $$\tag{1} A^T(Ax-b) = 0$$ or equivalently $$A^TA x = A^T b.$$ (This system of equations is often called the "normal equations". Visually, equation (1) just says that the residual vector $$b - Ax$$ is orthogonal to the column space of $$A$$.)

It follows that $$x = (A^T A)^{-1} A^T b$$. So the projection of $$b$$ onto $$S$$ is $$P(x) = Ax = A(A^T A)^{-1} A^T b.$$

• So it isn't necessarily true that all projection matrices take that form? – Kid Cudi Jan 1 at 13:05