Showing Orthogonal Projection Matrix Multiplied by Full-Rank Matrices is Positive-Definite Here is some useful information pertaining to my question:


*

*Let $X \in\mathbb{R}^{n \times m}$ and $Z \in\mathbb{R}^{n \times p}$ be full-rank matrices.

*Define $B = I_n - X(X^{T}X)^{-1}X^{T}$.

*Assume that the columns of $X$ are linearly independent from the columns of $Z$.


I am trying to show that $Z^{T}BZ$ is positive definite. 
My first plan of attack was to (1) show that $Z^{T}BZ$ is an orthogonal projection matrix, (2) prove that $I$ is the only positive definite orthogonal projection matrix, and (3) prove that $Z^{T}BZ=I$. This fell through because I felt that there was not enough information about $Z$ to prove (1).
I am pretty sure that I will ultimately need to show either (i) $z^{T}Z^{T}BZz>0$ $\forall$ $z\in \mathbb{R}^p$ OR (ii) all the eigenvalues of $Z^{T}BZ$ are positive. I think my biggest problem stems from not knowing how to deal with $Z^{T}$ and $Z$... clearly they are important, as it seems that we can only prove that $B$ is positive semidefinite. However, I am unsure how this full-rank matrix can "transform" $B$ from psd to pd.
Any assistance would be greatly appreciated! If it helps, I have already shown that B is an orthogonal projection matrix and that B is psd. :)
 A: Hmm, I am a bit surprised that no one went and finished the proof.
So how do we prove that $ ( I - X ( X^T X )^{-1} X^T ) Z $ has linearly independent columns?  Assume that it doesn't and show that this results in a contradiction.  We will look at $ Z - X ( X^T X )^{-1} X^T  Z $ instead.
Let $ x $ be such that $ (Z - X ( X^T X )^{-1} X^T  Z) x = 0 $ 


*

*Maybe $ Z x = 0 $.  But that can't be because $ Z $ has linearly independent columns.  So, $ Z x \neq 0 $.

*Maybe $ X^T ( Z x ) = 0 $.  That can't be since then $ Z x - X ( X^T X )^{-1} X^T  Z x = Z x = 0 $ because $ Z $ has linearly independent columns.  So, $  X^T Z x \neq 0 $.  This also means $ y = ( X^T X )^{-1} X^T  Z x \neq 0 $.
This reasoning tells us that  $ Z x - X y = 0 $ or, equivalently, $ Z x = X y $ for both $ x \neq 0 $ and $ y \neq 0 $.  But that in turn means that the columns of $ Z $ are not linearly independent of the columns of $ X $ since then $$ \left( \begin{array}{c | c} Z & X \end{array} \right) \left( \begin{array}{c} x\\-y \end{array} \right) = 0. $$
A: This is an interesting question.
Some observations:  


*

*$ I - X ( X^T X)^{-1} X^T $ projects onto the subspace orthogonal to the column space of $ X $.

*Applying a projection twice vs. once does not change the result.

*A projection is symmetric.
So, $Z^T B Z = Z^T B B Z = Z^T B^T B Z = ( B Z )^T B Z $.
Thus, the equivalent question is to prove that $ B Z $ has linearly independent columns.  ($ A^T A $ is positive definite if and only if $ A $ has linearly independent columns.)
Intuitively, $ B Z $ should have linearly independent columns since each column of $ Z$ needs to have a component orthogonal to the column space of $ X $ or else it is in the column space of $ X $.
I'll let someone else formalize the last observation.
A: To prove positive definite, we must show that $x^TZ^TBZx=0$ implies $x=0$. We have $B^TB=B$, since $B$ is symmetric and idempotent. Hence $x^TZ^TBZx=(BZx)^T(BZx)=\|BZx\|^2$ and therefore $BZx=0$.
Substituting $B:=I-X(X^TX)^{-1}X^T$, this means $Zx-Xy=0$, where $y:=(X^TX)^{-1}X^TZx.$ Since the columns of $Z$ are linearly independent of the columns of $X$, this implies that $Zx=0$ or $Xy=0$.

*

*If $Zx=0$, this implies $x=0$ since the columns of $Z$ are linearly independent.

*If $Xy=0$, then $Zx=Zx-Xy=0$ so again $x=0$.

