Transpose of a projection matrix The projection matrix for some matrix $A$, which has $m>n$ is:
$$P = A(A^TA)^{-1}A^T$$
A property of the projection matrix is that: $$P^T=P$$
If I call $(A^TA)^{-1}=B$, then:
$$P=(AB)A^T$$
Now I take transpose: $$P^T = ((AB)A^T)^T=A(AB)^T=AB^TA^T$$ or $$P^T=A((A^TA)^{-1})^TA^T$$
And is this equal to $P$? That middle term is transposed...
 A: Remember that transposition and inversion commute, i.e. the transpose of the inverse is equal to the inverse of the transpose:
$$\left(B^\mathrm{T}\right)^{-1} = \left(B^{-1}\right)^\mathrm{T}.$$
Using this fact, we have
$$\left(\left(A^\mathrm{T}A\right)^{-1}\right)^\mathrm{T} = \left(\left(A^\mathrm{T}A\right)^{\mathrm{T}}\right)^{-1} = \left(A^\mathrm{T}A\right)^{-1},$$
where the last equality follows since $A^\mathrm{T}A$ is symmetric.
A: The inverse and the transpose operations commute: assuming $A$ is square, $(A^T)^{-1}=(A^{-1})^T$, because
$$A^T((A^{-1})^T)=(A^{-1}A)^T=I^T=I$$
A: Yes, $P^T$ is equal to $P$ because $(A^T A)^{-1}$ is equal to its transpose. This is because for any invertible matrix $X$, $(X^T)^{-1} = (X^{-1})^T$, ie the operations of taking inverses and transposes commute.  So we have that the transpose $((A^TA)^{-1})^T = ((A^TA)^T)^{-1}$, which equals $ (A^T A)^{-1}$ because $(A^TA)^T = A^TA$. 
To prove that taking inverses and transposes can be done in either order, suppose $X$ is a matrix with inverse $X^{-1}$, so that $X X^{-1}=I$. Taking the transpose of both sides, we get $(X^{-1})^TX^T=I$.  In other words, $(X^{-1})^T$ is the inverse of $X^T$, i.e. $(X^{-1})^T = (X^T)^{-1}$.
A: If $A$ has full column rank, then its SVD is of the form
$$A = U \Sigma V^T = \begin{bmatrix} U_1 & U_2\end{bmatrix} \begin{bmatrix} \hat{\Sigma}\\ O\end{bmatrix} V^T$$
Thus,
$$A^T A = V \Sigma^T U^T U \Sigma V^T = V \Sigma^T \Sigma V^T = V \hat{\Sigma}^2 V^T$$
and
$$(A^T A)^{-1} = V \hat{\Sigma}^{-2} V^T$$
Hence,
$$A (A^T A)^{-1} A^T = U \Sigma V^T V \hat{\Sigma}^{-2} V^T V \Sigma^T U^T = U \Sigma \hat{\Sigma}^{-2} \Sigma^T U^T = U \begin{bmatrix} I & O\\ O & O\end{bmatrix} U^T = U_1 U_1^T$$
which is symmetric. The columns of $U_1$ form an orthonormal basis for the column space of $A$.
