Here is some useful information pertaining to my question:

  • Let $X \in\mathbb{R}^{n \times m}$ and $Z \in\mathbb{R}^{n \times p}$ be full-rank matrices.
  • Define $B = I_n - X(X^{T}X)^{-1}X^{T}$.
  • Assume that the columns of $X$ are linearly independent from the columns of $Z$.

I am trying to show that $Z^{T}BZ$ is positive definite.

My first plan of attack was to (1) show that $Z^{T}BZ$ is an orthogonal projection matrix, (2) prove that $I$ is the only positive definite orthogonal projection matrix, and (3) prove that $Z^{T}BZ=I$. This fell through because I felt that there was not enough information about $Z$ to prove (1).

I am pretty sure that I will ultimately need to show either (i) $z^{T}Z^{T}BZz>0$ $\forall$ $z\in \mathbb{R}^p$ OR (ii) all the eigenvalues of $Z^{T}BZ$ are positive. I think my biggest problem stems from not knowing how to deal with $Z^{T}$ and $Z$... clearly they are important, as it seems that we can only prove that $B$ is positive semidefinite. However, I am unsure how this full-rank matrix can "transform" $B$ from psd to pd.

Any assistance would be greatly appreciated! If it helps, I have already shown that B is an orthogonal projection matrix and that B is psd. :)


Hmm, I am a bit surprised that no one went and finished the proof.

So how do we prove that $ ( I - X ( X^T X )^{-1} X^T ) Z $ has linearly independent columns? Assume that it doesn't and show that this results in a contradiction. We will look at $ Z - X ( X^T X )^{-1} X^T Z $ instead.

Let $ x $ be such that $ (Z - X ( X^T X )^{-1} X^T Z) x = 0 $

  • Maybe $ Z x = 0 $. But that can't be because $ Z $ has linearly independent columns. So, $ Z x \neq 0 $.

  • Maybe $ X^T ( Z x ) = 0 $. That can't be since then $ Z x - X ( X^T X )^{-1} X^T Z x = Z x = 0 $ because $ Z $ has linearly independent columns. So, $ X^T Z x \neq 0 $. This also means $ y = ( X^T X )^{-1} X^T Z x \neq 0 $.

This reasoning tells us that $ Z x - X y = 0 $ or, equivalently, $ Z x = X y $ for both $ x \neq 0 $ and $ y \neq 0 $. But that in turn means that the columns of $ Z $ are not linearly independent of the columns of $ X $ since then $$ \left( \begin{array}{c | c} Z & X \end{array} \right) \left( \begin{array}{c} x\\-y \end{array} \right) = 0. $$

  • $\begingroup$ Thanks so much for responding! I ultimately ended up showing similar to this, but you wrote it so much more clearly. $\endgroup$ – Benjamooky Apr 3 '18 at 16:31
  • $\begingroup$ This was a fun question. $\endgroup$ – Robert van de Geijn Apr 3 '18 at 18:59

This is an interesting question.

Some observations:

  • $ I - X ( X^T X)^{-1} X^T $ projects onto the subspace orthogonal to the column space of $ X $.

  • Applying a projection twice vs. once does not change the result.

  • A projection is symmetric.

So, $Z^T B Z = Z^T B B Z = Z^T B^T B Z = ( B Z )^T B Z $.

Thus, the equivalent question is to prove that $ B Z $ has linearly independent columns. ($ A^T A $ is positive definite if and only if $ A $ has linearly independent columns.)

Intuitively, $ B Z $ should have linearly independent columns since each column of $ Z$ needs to have a component orthogonal to the column space of $ X $ or else it is in the column space of $ X $.

I'll let someone else formalize the last observation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.