Suppose $K^TX=0$ and $K$ has full column rank. Suppose furthermore that the columns of $X$ form a basis for the nullspace of $K^T$. If $H$ is positive definite, then

$$ K(K^THK)^{-1}K^T = H^{-1} − H^{-1}X(X^TH^{-1}X)^{-1}X^TH^{-1} $$

The identity can be found on slide 44 at http://www.ltcc.ac.uk/media/qmul-images/REML-Lecture-1.pdf. A quick R script has also verified the formula. However, I can't seem to prove the identity. I have tried using the Schur formula for inverses, but it does not seem to help in this case.

  • $\begingroup$ You need to specify that the columns of $X$ give a basis for the nullspace of $K^\top$. Then you should see why the formula holds when $H$ is the identity. $\endgroup$ Dec 16, 2022 at 5:22
  • $\begingroup$ @TedShifrin Thank you Prof. Shifrin. That makes perfect sense now: In the case where $H=I$, then for any vector $v$ we may express $v$ uniquely as $v=K \gamma_1 + X\gamma_2$. Then, $K(K^TK)^{-1}K^T(K \gamma_1 + X\gamma_2) + X(X^TX)^{-1}X^T(K \gamma_1 + X\gamma_2) = K \gamma_1 + X\gamma_2$ is straightforward to see which yields the formula $K(K^TK)^{-1}K^T + X(X^TX)^{-1}X^T = I$. When $H \neq I$, if we could somehow express $v=HK \gamma_1 + X \gamma_2$, we could then use the same strategy to show the formula holds - but it is not obvious to me that $v=HK \gamma_1 + X \gamma_2$ $\endgroup$
    – Abm
    Dec 16, 2022 at 7:46

1 Answer 1


Although you have not revised appropriately, we agree that the columns of $X$ give a basis for the nullspace of $K^\top$. This subspace is the orthogonal complement of the column space $W$ of $K$. Assuming $H=I$, we therefore have \begin{align*} I &= \text{proj}_W + \text{proj}_{W^\perp} \\ &= K(K^\top K)^{-1}K^\top + X(X^\top X)^{-1}X^\top. \end{align*} Here $\text{proj}_V$ is the matrix for the orthogonal projection onto the subspace $V$.

In the general case, we let $\tilde K = \sqrt H K$, where $\sqrt H$ is the unique positive square root of the positive definite symmetric matrix $H$. We set $\tilde X = \sqrt H^{-1} X$. Then the columns of $\tilde X$ give a basis for the nullspace of $\tilde K$. Thus, as before, we have \begin{align*} I &= \tilde K(\tilde K^\top\tilde K)^{-1}\tilde K^\top + \tilde X(\tilde X^\top\tilde X)^{-1}\tilde X^\top \\ &= \sqrt H K(K^\top H K)K^\top \sqrt H + \sqrt H^{-1}X^\top(X^\top H^{-1}X)^{-1}X^\top\sqrt H^{-1}, \text{so} \\ H^{-1} &= K(K^\top HK)K^\top + H^{-1}X^\top(X^\top H^{-1}X)^{-1}X^\top H^{-1}, \end{align*} as desired.

  • $\begingroup$ Thank you again, Prof. Shifrin, fully understood. I did not think of forming the matrices $\bar{K}$ and $\bar{X}$ - that was nice. I will update my question to include the correct assumptions on $X$ for future readers $\endgroup$
    – Abm
    Dec 17, 2022 at 7:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .