How to find the singular value decomposition of $A^TA$ & $(A^TA)^{-1}$ I want to find the singular value decomposition of $A^TA$ & $(A^TA)^{-1}$.
The singular value decomposition of $A$ is $$A=U \Sigma V^T$$
Basically, I want to find the singular values of $A^TA$ & $(A^TA)^{-1}$
$$A^TA=V{\Sigma}^2 V^T$$.
Does this mean that the singular values of $A^TA$ is equal to square of the singular values of $A$?
How to find the $(A^TA)^{-1}$?
 A: Finding the singular value decomposition
Start with a matrix with $m$ rows, $n$ columns, and rank $\rho$:
$$
  \mathbf{A} = \mathbb{C}^{m \times n}_{\rho}.
$$
Since the question concerns the normal equations, let's fix $\rho = m$ and $m\ge n$. The matrix $\mathbf{A}$ is tall and has full column rank.
The singular value decomposition is
$$
\begin{align}
  \mathbf{A} &= \mathbf{U}\, \mathbf{\Sigma} \, \mathbf{V}^{*} \\
  &=
 \left[ \begin{array}{cc}
   \color{blue}{\mathbf{U}_{\mathcal{R}\left( \mathbf{A} \right)}} &
   \color{red}{\mathbf{U}_{\mathcal{N}\left( \mathbf{A}^{*} \right)}}
 \end{array} \right]
%
  \left[ \begin{array}{c}
    \mathbf{S} \\ \mathbf{0}
  \end{array} \right] 
%
   \color{blue}{\mathbf{V}_{\mathcal{R}\left( \mathbf{A}^{*} \right)}}^{*} 
%
\end{align}
$$
The coloring distinguishes $\color{blue}{range}$ spaces from $\color{red}{null}$ spaces.
The diagonal matrix of singular values, $\mathbf{S}\in\mathbb{R}^{\rho\times\rho}$ is
$$
  \mathbf{S}_{k,k} = \sigma_{k}, \quad k=1,\rho.
$$
Manipulating the singular value decomposition
The Moore-Penrose pseudoinverse is
$$
\begin{align}
  \mathbf{A}^{\dagger} 
     &= \mathbf{V}\, \Sigma^{\dagger} \mathbf{U}^{*} \\
%
  &=
%
  \color{blue}{\mathbf{V}_{\mathcal{R}\left( \mathbf{A}^{*} \right)}}
%
  \left[ \begin{array}{cc}
    \mathbf{S}^{-1} & \mathbf{0}
  \end{array} \right] 
%
  \left[ \begin{array}{l}
    \color{blue}{\mathbf{U}_{\mathcal{R} \left( \mathbf{A} \right)}}^{*} \\   
    \color{red}{\mathbf{U}_{\mathcal{N} \left( \mathbf{A}^{*} \right)}}^{*}
  \end{array} \right].
%
\end{align}
$$
The Hermitian conjugate is
$$
\begin{align}
  \mathbf{A}^{*} 
     &= \mathbf{V}\, \Sigma^{\mathrm{T}} \mathbf{U}^{*} \\
%
  &=
%
  \color{blue}{\mathbf{V}_{\mathcal{R}\left( \mathbf{A}^{*} \right)}}
%
  \left[ \begin{array}{cc}
    \mathbf{S} & \mathbf{0}
  \end{array} \right] 
%
  \left[ \begin{array}{l}
    \color{blue}{\mathbf{U}_{\mathcal{R} \left( \mathbf{A} \right)}}^{*} \\   
    \color{red}{\mathbf{U}_{\mathcal{N} \left( \mathbf{A}^{*} \right)}}^{*}
  \end{array} \right].
%
\end{align}
$$
Resolving the product matrix
The product matrix has a simple expression:
$$
\begin{align}
  \mathbf{A}^{*} \mathbf{A} &= 
  \left( \mathbf{V} \, \Sigma^{\mathrm{T}} \mathbf{U}^{*} \right)
  \left( \mathbf{U} \, \Sigma \mathbf{V}^{*} \right) \\
%
 &=
%
  \color{blue}{\mathbf{V}_{\mathcal{R}\left( \mathbf{A}^{*} \right)}}
  \, \mathbf{S}^{2} \,
  \color{blue}{\mathbf{V}_{\mathcal{R}\left( \mathbf{A}^{*} \right)}}^{*}.
%
\end{align}
$$
The pseudoinverse of the product matrix is then
$$
\begin{align}
  \left( \mathbf{A}^{*} \mathbf{A} \right)^{-1} 
 &=
%
  \left(
  \color{blue}{\mathbf{V}_{\mathcal{R}\left( \mathbf{A}^{*} \right)}}
  \, \mathbf{S}^{-2} \,
  \color{blue}{\mathbf{V}_{\mathcal{R}\left( \mathbf{A}^{*} \right)}}^{*}
  \right)^{-1} \\[3pt]
%
  &=
%
  \left(
    \color{blue}{\mathbf{V}_{\mathcal{R}\left( \mathbf{A}^{*} \right)}}^{*}
  \right)^{-1}
  \left( \mathbf{S}^{2} \right)^{-1}
  \left(
  \color{blue}{\mathbf{V}_{\mathcal{R}\left( \mathbf{A}^{*} \right)}}
  \right)^{-1} \\[3pt]
%
 &=
%
  \color{blue}{\mathbf{V}_{\mathcal{R}\left( \mathbf{A}^{*} \right)}}
  \, \mathbf{S}^{-2} \,
  \color{blue}{\mathbf{V}_{\mathcal{R}\left( \mathbf{A}^{*} \right)}}^{*}.
%
\end{align}
$$
