Consider a linear regression $$ y=X_1\beta_1+X_2\beta_2+u $$ to be estimated by least squares. Here, $y$, $X_1$, and $X_2$ are $n\times 1$, $n\times k_1$ and $n\times k_2$. Let $X=(X_1,X_2)$ to be $n\times k$ where $k=k_1+k_2$. Typically, one assumes $X$ is of full column rank.

It is well known that:

  • The least squares estimator for $\beta=\begin{pmatrix}\beta_1\\\beta_2\end{pmatrix}$ is given by $\hat\beta\equiv(X'X)^{-1}X'y$. Since $X$ has rank $k$, $X'X$ is invertible so $\hat\beta$ is well-defined.
  • The least squares estimator for $\beta_1$ can be written as $\hat\beta_1\equiv(X_1'M_2X_1)^{-1}X_1'M_2y$ where $M_2=I_n-X_2(X_2'X_2)^{-1}X_2'$. (For example, see here).

Since $\hat\beta_1$ is a subvector of $\hat\beta$, $\hat\beta_1$ is obviously well-defined. My questions are:

  1. How do we see directly that $X_1'M_2X_1$ is invertible from $X$ having full column rank?
  2. Is $X_1'M_2X_1$ still invertible if we start with assuming $X_1$ has rank $k_1$ only ($M_2$ will have to be redefined as $I_n-X_2(X_2'X_2)^-X_2'$)? Obviously, something beyond $X_1$ having rank $k_1$ is necessary (if $X_2=X_1$, then $X_1'M_2X_1=0$), so what is that "something" please?

$X'X$ is symmetric and is positive semi-definite by definition, which does not yet guarantee invertibility, since $0$ is a possible eigenvalue. To guarantee invertibility one might simply do:

$$X'X = X'X + cI$$

where $c>0$ is a very small number and I is the identity matrix. $X'X + cI$ is now guaranteed to be positive-definite and therefore invertible. For small problems you might now use the Cholesky-Decomposition, for big problems some quadratic optimization algorithm.

  • $\begingroup$ I'm sorry but this answer is not useful. By the assumption that $X$ has rank $k$, $X'X$ is invertible and this can be shown by elementary means: if $\gamma\neq 0$ were such that $X'X\gamma=0$, then $0=\gamma'X'X\gamma=||X\gamma||^2$ and so $X\gamma=0$, a contradiction. In any case, the invertibility of $X'X$ is not what I'm asking. $\endgroup$ – yurnero Sep 23 '19 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.