# Proof projection is invariant to the choice of basis

Assume first $A$ has independent columns. We have a projection matrix $P$ on the range of $A$, i.e. $P_A = A(A^\top A)^{-1}A^\top$. Consider now a matrix $B$ which has the same range as $A$. We have $B = AC$, where $C$ is an invertible matrix. We also have the following.

$P_B = B(B^\top B)^{-1}B^\top = AC(C^\top A^\top AC )^{-1}C^\top A^\top = ACC^{-1}(A^\top A)^{-1} {C^{\top}}^{-1}C^\top A = P_A$

I am interested in having a similar result for the case where A does not have independent columns. Now, $A^\top A$ isn't invertible and the projection matrix becomes $P_A = A(A^\top A)^{\dagger}A^\top$, where we denote by $(\cdot)^\dagger$ the Moore-Penrose pseudoinverse. Consider now two cases:

1. $B$ also has dependent columns and $B = AC$ for some invertible $C$.
2. $B$ has independent columns and $B = AC$ for a skinny rectangular matrix $C$ such that $C^\top C = I$, i.e. $C$ selects a minimal subset of columns of $A$ to span the range. Edit: The columns of C are columns of the identity matrix.

In each case, I would like a proof that $A(A^\top A)^{\dagger}A^\top = B(B^\top B)^{\dagger}B^\top$. The problem with the above technique is that I can't simply pull $C$ out of $(C^\top A^\top AC )^{\dagger}$.

Edit: I see that the matrix $A(A^\top A)^{\dagger}A^\top = A A^\dagger$, but again, I don't see how to show that $A A^\dagger = AC (AC)^\dagger$.

Edit: I can also see the following justification: $P_A$ is the projection matrix on the range of $A$, and $P_B$ on the range of $B$. Since the range is the same, they also have to be the same by uniqueness of projection (@martini). But this is ugly, because we have to interpret the matrices as projections. Isn't there an $algebraic$ way to justify this?

-
For the case where the columns are not independent, I dont'think $P_A=A(A^{\top} A)^{\dagger} A^{\top}$. Example: if $A=\matrix{$$1 & 1$$}$, $A(A^{\top} A)^{\dagger} A^{\top}=(4)$, and this is not a projection. – francis-jamet Oct 4 '12 at 13:34
Don't we have $P_A = AA^\dagger$? This has the same range as $A$ and fullfills $AA^\dagger AA^\dagger = AA^\dagger$ and $(AA^\dagger)^\top = AA^\dagger$. – martini Oct 4 '12 at 13:37
Doesn't all this follow from the uniqueness of projectors? – martini Oct 4 '12 at 13:41
$A(A^{\mathsf{T}}A)^{-1}A^{\mathsf{T}}=I$, so I'm not sure you meant that. – Matthew Pressland Oct 4 '12 at 13:43
@MattPressland: For the case where the columns are independent, $A$ is not necessarily a square matrix. – francis-jamet Oct 4 '12 at 13:51

If $A^{\top}$ is the transpose of $A$, we may have $A \neq 0$ and $P_A=0$. So, $P_A$ is not the projection on the range of $A$ in this case.

Example: If $A=\left( \matrix {1 & 1+i \\ i &i-1} \right)$, $A^{\top} A=0$, so $(A^{\top}A)^{\dagger}=0$ and $P_A=0$, but $A \neq0$.

So, we suppose $A^{\top}$ is the Hermitian transpose (or conjugate transpose). We have $P_A P_A=P_A$ and $P_A^{\top}=P_A$.

1. If $B=AC$ with $C$ invertible. We have $A^{\top}A(A^{\top}A)^{\dagger}A^{\top}A=A^{\top}A$. So, $$P_B P_A P_B = B(B^{\top}B)^{\dagger}C^{\top}A^{\top}P_A AC (B^{\top}B)^{\dagger}B$$ $$=B(B^{\top}B)^{\dagger}C^{\top}A^{\top}AC (B^{\top}B)^{\dagger}B=P_B P_B=P_B$$

As $A=BC^{-1}$, we have $P_A P_B P_A=P_A$ too.

If $M$ is a matrix, $M^{\top}M=0 \implies M=0$.

If $M=(Id-P_B)P_A$, $M^{\top}M=P_A(Id-P_B)P_A=P_A-P_A P_BP_A=0$, so $M=0$ and $P_A-P_BP_A=0$, and $P_A=P_B P_A$. By hermitian transposition, $P_A=P_A P_B$.

We have $P_B=P_A P_B=P_B P_A$, so $P_A=P_B$.

-
Thank you for the excellent answer. Actually, this also solves point 2, since invertability of $C$ isn't required. Because the span of $A$ and $B$ is the same, we can write $A(A^\top A)^\dagger A^\top = ACX = BX$ for some $X$. Then $P_A P_B P_A = X^\top B^\top B (B^\top B)^\dagger B^\top B X = X^\top B^\top B X = P_A P_A = P_A$. – ziutek Oct 8 '12 at 16:34