In least-squares, what if ${\bf X}^\top {\bf X}$ is not invertible? In least-squares, when solving the normal equation, we calculate the inverse of ${\bf X}^\top {\bf X}$. What if this matrix is not invertible?
 A: Start with the linear system
$$
\mathbf{A}x = b
$$
with
$$
  \mathbf{A} \in \mathbb{C}^{m\times n}_{\rho}, \quad
  x \in \mathbb{C}^{n}, \quad
  b \in \mathbb{C}^{m}.
$$
If the data vector $b$ is in the image of $\mathbf{A}$, then there is a solution vector $x$ such that
$$
 \mathbf{A}x - b = 0.
$$
If the data vector $b$ has a nullspace component, we can not get an exact answer, and instead ask for the best answer.
Choosing the $2-$norm, the least squares minimizers are defined as
$$
x_{LS} = \left\{ x\in\mathbb{C}^{n} \colon \lVert \mathbf{A} x_{LS} - b \rVert_{2}^{2} \text{ is minimized} \right\}.
$$
Every matrix has a singular value decomposition
$$
\mathbf{A} = \mathbf{U}\, \Sigma\, \mathbf{V}^{*}
$$
which allows us to express the Moore-Penrose pseudoinverse
$$
\mathbf{A}^{\dagger} = \mathbf{V}\, \Sigma^{\dagger}\, \mathbf{U}^{*}
$$
which can be used to pose the general solution to the least squares problem:
$$
x_{LS} = \mathbf{A}^{\dagger} b + \left( \mathbf{I}_{n} - \mathbf{A}^{\dagger}\mathbf{A} \right) y, \quad y\in\mathbb{C}^{n}
$$
The geometry of the solution is discussed in Is the unique least norm solution to Ax=b the orthogonal projection of b onto R(A)?
More on the SVD here: How does the SVD solve the least squares problem?
When the classic inverse exists, it is the pseudoinverse. In this case, there are no nullspace components and $\Sigma=\mathbf{S}$ and we can see that
$$
 \begin{align}
%
  \mathbf{A}\mathbf{A}^{-1} &= \mathbf{A}\mathbf{A}^{\dagger} = 
\left( \mathbf{V}\, \mathbf{S}\, \mathbf{U}^{*}\right)
\left( \mathbf{U}\, \mathbf{S}^{-1}\, \mathbf{V}^{*}\right) = \mathbf{I} \\
%
  \mathbf{A}^{-1}\mathbf{A} &= \mathbf{A}^{\dagger}\mathbf{A} = 
\left( \mathbf{U}\, \mathbf{S}^{-1}\, \mathbf{V}^{*}\right)
\left( \mathbf{V}\, \mathbf{S}\, \mathbf{U}^{*}\right) = \mathbf{I}
%
 \end{align}
$$
The normal equations solution is discussed here: Difference between orthogonal projection and least squares solution
Additional discussion here: Least squares and pseudo-inverse
