# Invertibility of a matrix that arises from least squares estimation

Consider a linear regression $$y=X_1\beta_1+X_2\beta_2+u$$ to be estimated by least squares. Here, $$y$$, $$X_1$$, and $$X_2$$ are $$n\times 1$$, $$n\times k_1$$ and $$n\times k_2$$. Let $$X=(X_1,X_2)$$ to be $$n\times k$$ where $$k=k_1+k_2$$. Typically, one assumes $$X$$ is of full column rank.

It is well known that:

• The least squares estimator for $$\beta=\begin{pmatrix}\beta_1\\\beta_2\end{pmatrix}$$ is given by $$\hat\beta\equiv(X'X)^{-1}X'y$$. Since $$X$$ has rank $$k$$, $$X'X$$ is invertible so $$\hat\beta$$ is well-defined.
• The least squares estimator for $$\beta_1$$ can be written as $$\hat\beta_1\equiv(X_1'M_2X_1)^{-1}X_1'M_2y$$ where $$M_2=I_n-X_2(X_2'X_2)^{-1}X_2'$$. (For example, see here).

Since $$\hat\beta_1$$ is a subvector of $$\hat\beta$$, $$\hat\beta_1$$ is obviously well-defined. My questions are:

1. How do we see directly that $$X_1'M_2X_1$$ is invertible from $$X$$ having full column rank?
2. Is $$X_1'M_2X_1$$ still invertible if we start with assuming $$X_1$$ has rank $$k_1$$ only ($$M_2$$ will have to be redefined as $$I_n-X_2(X_2'X_2)^-X_2'$$)? Obviously, something beyond $$X_1$$ having rank $$k_1$$ is necessary (if $$X_2=X_1$$, then $$X_1'M_2X_1=0$$), so what is that "something" please?

$$X'X$$ is symmetric and is positive semi-definite by definition, which does not yet guarantee invertibility, since $$0$$ is a possible eigenvalue. To guarantee invertibility one might simply do:
$$X'X = X'X + cI$$
where $$c>0$$ is a very small number and I is the identity matrix. $$X'X + cI$$ is now guaranteed to be positive-definite and therefore invertible. For small problems you might now use the Cholesky-Decomposition, for big problems some quadratic optimization algorithm.
• I'm sorry but this answer is not useful. By the assumption that $X$ has rank $k$, $X'X$ is invertible and this can be shown by elementary means: if $\gamma\neq 0$ were such that $X'X\gamma=0$, then $0=\gamma'X'X\gamma=||X\gamma||^2$ and so $X\gamma=0$, a contradiction. In any case, the invertibility of $X'X$ is not what I'm asking. – yurnero Sep 23 '19 at 17:37