When is Block-Partitioned Matrix Invertible?

Suppose I have a block partitioned matrix $$\begin{bmatrix} \mathbf{X}_1^{\top}\mathbf{X}_1 & \mathbf{X}_1^{\top}\mathbf{X}_2 \\ \mathbf{X}_2^{\top}\mathbf{X}_1 & \mathbf{X}_2^{\top}\mathbf{X}_2 \\ \end{bmatrix}$$ where $\mathbf{X}_1$ is $G \times K_1$ and $\mathbf{X}_2$ is $G \times K_2$. Thus our block partitioned matrix is equal to the outer product $\begin{bmatrix} \mathbf{X}_1^{\top} \\ \mathbf{X}_2^{\top} \end{bmatrix}\begin{bmatrix} \mathbf{X}_1 & \mathbf{X}_2 \end{bmatrix}$ and is dimension $K \times K$ where $K=K_1+K_2$.

In general, when is this block-partitioned matrix invertible? Is there a necessary and sufficient condition?

Edit. My first thought is: for the block-partitioned matrix to be invertible, it is equivalent that each of the four blocks are invertible. However, I no longer think this. For example, if one of the off diagonals, say the lower left, $\mathbf{X}_2^{\top}\mathbf{X}_1=\mathbf{0}$, the matrix could still be invertible if the blocks on the diagonal are. I can't articulate a proof of that though -- could anyone help?

Thanks!

The matrix $$Y:=X^TX, \quad X:=[X_1,X_2],$$ (which is generally positive semidefinite) is invertible iff $[X_1,X_2]$ has full column rank.
So necessarily, $X_1$ must have full column rank. However, full rank of $X_2$ is not sufficient for the nonsingularity of $Y$. From the block inversion formulas it follows that
$X$ is invertible iff $X_1$ has full column rank and the Schur complement $$\tag{1}X_2^TX_2-X_2^TX_1(X_1^TX_1)^{-1}X_1^TX_2=X_2^T(I-X_1(X_1^TX_1)^{-1}X_1^T)X_2$$ is invertible.
The matrix $P_1:=I-X_1(X_1^TX_1)^{-1}X_1^T$ is an orthogonal projector onto the complement of the range of $X_1$, that is, the nullspace of $X_1^T$. Since $X_2^T(I-X_1(X_1^TX_1)^{-1}X_1^T)X_2=(P_1X_1)^T(P_1X_1)$, we have that (1) is invertible iff $P_1X_1$ has full column rank, that is, the columns of $X_1$ do not become dependent when projected onto the nullspace of $X_1^T$.
Note that the same argument can be made with the indices $1$ and $2$ exchanged.