What is the easiest way to find the inverse of the following block diagonal matrix? Consider, for $\sigma^2_u, \sigma^2_e > 0$, the matrix
$$\mathbf{X} = \begin{bmatrix}
\sigma^2_u+\sigma^2_e & & & \\
 & \sigma^2_u+\sigma^2_e\ \\
& & \sigma^2_u+\sigma^2_e & \sigma^2_u \\
& & \sigma^2_u & \sigma^2_u+\sigma^2_e
\end{bmatrix}$$
where all other entries are $0$. What is the easiest way to find the inverse of $\mathbf{X}$?
I thought about adjoining $\mathbf{X}$ to $\mathbf{I}_4$: $$\left[\begin{array}{cccc}
\sigma^2_u+\sigma^2_e & & & \\
 & \sigma^2_u+\sigma^2_e\ \\
& & \sigma^2_u+\sigma^2_e & \sigma^2_u \\
& & \sigma^2_u & \sigma^2_u+\sigma^2_e\end{array}\right|\left.\begin{array}{cccc}
1 &  & & & \\
  & 1 & & \\
  &   & 1 & \\
  &   &   & 1 
\end{array}\right]$$
and thought about dividing each row by $\sigma^2_u+\sigma^2_e =: K$:
$$\left[\begin{array}{cccc}
1 & & & \\
 & 1\ \\
& & 1& \sigma^2_u/K \\
& & \sigma^2_u/K & 1\end{array}\right|\left.\begin{array}{cccc}
1/K &  & & & \\
  & 1/K & & \\
  &   & 1/K & \\
  &   &   & 1/K 
\end{array}\right]$$
but I'm not sure how to proceed from here.
I am well-aware of this formula but would like to avoid it.
 A: You got
$$\left[\begin{array}{cccc}
1 & & & \\
 & 1\ \\
& & 1& \sigma^2_u/K \\
& & \sigma^2_u/K & 1\end{array}\right|\left.\begin{array}{cccc}
1/K &  & & & \\
  & 1/K & & \\
  &   & 1/K & \\
  &   &   & 1/K 
\end{array}\right]$$
Now do the row operation $Row_4\leftarrow Row_4-\sigma^2_u/K Row_3$:
$$\left[\begin{array}{cccc}
1 & & & \\
 & 1\ \\
& & 1& \sigma^2_u/K \\
& & 0 & 1-(\sigma^2_u/K)^2\end{array}\right|\left.\begin{array}{cccc}
1/K &  & & & \\
  & 1/K & & \\
  &   & 1/K & \\
  &   &  -\sigma^2_u/K^2 & 1/K 
\end{array}\right]$$
Divide the last row by $1-(\sigma^2_u/K)^2=1/\lambda$
$$\left[\begin{array}{cccc}
1 & & & \\
 & 1\ \\
& & 1& \sigma^2_u/K \\
& & 0 & 1\end{array}\right|\left.\begin{array}{cccc}
1/K &  & & & \\
  & 1/K & & \\
  &   & 1/K & \\
  &   &  -\lambda\sigma^2_u/K^2 & \lambda/K 
\end{array}\right]$$
Finally do the row operation $Row_3\leftarrow Row_3 - \sigma^2_u/KRow_4$
$$\left[\begin{array}{cccc}
1 & & & \\
 & 1\ \\
& & 1& 0 \\
& & 0 & 1\end{array}\right|\left.\begin{array}{cccc}
1/K &  & & & \\
  & 1/K & & \\
  &   & 1/K+\lambda\sigma^4_u/K^3 & -\lambda\sigma^2_u/K^2\\
  &   &  -\lambda\sigma^2_u/K^2 & \lambda/K 
\end{array}\right]$$
And you are done. Notice that even without doing the gaussian elimination on the full matrix, it would have been possible to anticipate that only the lower right corner would be modified. Hence it's equivalent to invert only this $2\times2$ matrix, and to invert the other diagonal elements separately. Or, if you prefer, to invert the blocks separately (there are two $1\times1$ blocks and one $2\times 2$).
A: If
$\mathbf{X} =
\begin{bmatrix}
      A & & & \\
      & B & & \\
      & & C & \\
      & & & D
\end{bmatrix}$ is a block-diagonal matrix, then
$\mathbf{X}^{-1} =
\begin{bmatrix}
        A^{-1} &        &        &        \\
               & B^{-1} &        &        \\
               &        & C^{-1} &        \\
               &        &        & D^{-1}
\end{bmatrix}$
So
$\mathbf{X}^{-1} =
\begin{bmatrix}
\dfrac{1}{\sigma^2_u+\sigma^2_e} & & &
\\
 & \dfrac{1}{\sigma^2_u+\sigma^2_e}
\\
 & &
     \dfrac{\sigma_u^2+\sigma_e^2}{\sigma_e^2(2\sigma_u^2+\sigma_e^2)}
 & -\dfrac{\sigma^2_u}{\sigma_e^2(2\sigma_u^2+\sigma_e^2)}
\\
 &
 & -\dfrac{\sigma^2_u}{\sigma_e^2(2\sigma_u^2+\sigma_e^2)}
 & \dfrac{\sigma_u^2+\sigma_e^2}{\sigma_e^2(2\sigma_u^2+\sigma_e^2)}
\end{bmatrix}$
where 
$\begin{bmatrix}
    \sigma^2_u+\sigma^2_e & \sigma^2_u \\
    \sigma^2_u & \sigma^2_u+\sigma^2_e
\end{bmatrix}^{-1}
=
\dfrac{1}{\sigma_e^2(2\sigma_u^2+\sigma_e^2)}
\begin{bmatrix}
    \sigma^2_u+\sigma^2_e & -\sigma^2_u \\
    -\sigma^2_u & \sigma^2_u+\sigma^2_e
\end{bmatrix}$
