Consider a linear regression $$ y=X_1\beta_1+X_2\beta_2+u $$ to be estimated by least squares. Here, $y$, $X_1$, and $X_2$ are $n\times 1$, $n\times k_1$ and $n\times k_2$. Let $X=(X_1,X_2)$ to be $n\times k$ where $k=k_1+k_2$. Typically, one assumes $X$ is of full column rank.

It is well known that:

  • The least squares estimator for $\beta=\begin{pmatrix}\beta_1\\\beta_2\end{pmatrix}$ is given by $\hat\beta\equiv(X'X)^{-1}X'y$. Since $X$ has rank $k$, $X'X$ is invertible so $\hat\beta$ is well-defined.
  • The least squares estimator for $\beta_1$ can be written as $\hat\beta_1\equiv(X_1'M_2X_1)^{-1}X_1'M_2y$ where $M_2=I_n-X_2(X_2'X_2)^{-1}X_2'$. (For example, see here).

Since $\hat\beta_1$ is a subvector of $\hat\beta$, $\hat\beta_1$ is obviously well-defined. My questions are:

  1. How do we see directly that $X_1'M_2X_1$ is invertible from $X$ having full column rank?
  2. Is $X_1'M_2X_1$ still invertible if we start with assuming $X_1$ has rank $k_1$ only ($M_2$ will have to be redefined as $I_n-X_2(X_2'X_2)^-X_2'$)? Obviously, something beyond $X_1$ having rank $k_1$ is necessary (if $X_2=X_1$, then $X_1'M_2X_1=0$), so what is that "something" please?

1 Answer 1


$X'X$ is symmetric and is positive semi-definite by definition, which does not yet guarantee invertibility, since $0$ is a possible eigenvalue. To guarantee invertibility one might simply do:

$$X'X = X'X + cI$$

where $c>0$ is a very small number and I is the identity matrix. $X'X + cI$ is now guaranteed to be positive-definite and therefore invertible. For small problems you might now use the Cholesky-Decomposition, for big problems some quadratic optimization algorithm.

  • $\begingroup$ I'm sorry but this answer is not useful. By the assumption that $X$ has rank $k$, $X'X$ is invertible and this can be shown by elementary means: if $\gamma\neq 0$ were such that $X'X\gamma=0$, then $0=\gamma'X'X\gamma=||X\gamma||^2$ and so $X\gamma=0$, a contradiction. In any case, the invertibility of $X'X$ is not what I'm asking. $\endgroup$
    – yurnero
    Commented Sep 23, 2019 at 17:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .