# Showing Orthogonal Projection Matrix Multiplied by Full-Rank Matrices is Positive-Definite

Here is some useful information pertaining to my question:

• Let $X \in\mathbb{R}^{n \times m}$ and $Z \in\mathbb{R}^{n \times p}$ be full-rank matrices.
• Define $B = I_n - X(X^{T}X)^{-1}X^{T}$.
• Assume that the columns of $X$ are linearly independent from the columns of $Z$.

I am trying to show that $Z^{T}BZ$ is positive definite.

My first plan of attack was to (1) show that $Z^{T}BZ$ is an orthogonal projection matrix, (2) prove that $I$ is the only positive definite orthogonal projection matrix, and (3) prove that $Z^{T}BZ=I$. This fell through because I felt that there was not enough information about $Z$ to prove (1).

I am pretty sure that I will ultimately need to show either (i) $z^{T}Z^{T}BZz>0$ $\forall$ $z\in \mathbb{R}^p$ OR (ii) all the eigenvalues of $Z^{T}BZ$ are positive. I think my biggest problem stems from not knowing how to deal with $Z^{T}$ and $Z$... clearly they are important, as it seems that we can only prove that $B$ is positive semidefinite. However, I am unsure how this full-rank matrix can "transform" $B$ from psd to pd.

Any assistance would be greatly appreciated! If it helps, I have already shown that B is an orthogonal projection matrix and that B is psd. :)

Hmm, I am a bit surprised that no one went and finished the proof.

So how do we prove that $( I - X ( X^T X )^{-1} X^T ) Z$ has linearly independent columns? Assume that it doesn't and show that this results in a contradiction. We will look at $Z - X ( X^T X )^{-1} X^T Z$ instead.

Let $x$ be such that $(Z - X ( X^T X )^{-1} X^T Z) x = 0$

• Maybe $Z x = 0$. But that can't be because $Z$ has linearly independent columns. So, $Z x \neq 0$.

• Maybe $X^T ( Z x ) = 0$. That can't be since then $Z x - X ( X^T X )^{-1} X^T Z x = Z x = 0$ because $Z$ has linearly independent columns. So, $X^T Z x \neq 0$. This also means $y = ( X^T X )^{-1} X^T Z x \neq 0$.

This reasoning tells us that $Z x - X y = 0$ or, equivalently, $Z x = X y$ for both $x \neq 0$ and $y \neq 0$. But that in turn means that the columns of $Z$ are not linearly independent of the columns of $X$ since then $$\left( \begin{array}{c | c} Z & X \end{array} \right) \left( \begin{array}{c} x\\-y \end{array} \right) = 0.$$

• Thanks so much for responding! I ultimately ended up showing similar to this, but you wrote it so much more clearly. – Benjamooky Apr 3 '18 at 16:31
• This was a fun question. – Robert van de Geijn Apr 3 '18 at 18:59

This is an interesting question.

Some observations:

• $I - X ( X^T X)^{-1} X^T$ projects onto the subspace orthogonal to the column space of $X$.

• Applying a projection twice vs. once does not change the result.

• A projection is symmetric.

So, $Z^T B Z = Z^T B B Z = Z^T B^T B Z = ( B Z )^T B Z$.

Thus, the equivalent question is to prove that $B Z$ has linearly independent columns. ($A^T A$ is positive definite if and only if $A$ has linearly independent columns.)

Intuitively, $B Z$ should have linearly independent columns since each column of $Z$ needs to have a component orthogonal to the column space of $X$ or else it is in the column space of $X$.

I'll let someone else formalize the last observation.