Singular Value Decomposition: Prove that singular values of A are square roots of eigenvalues of both $AA^{T}$ and $A^{T}A$. From Singular Value Decomposition, we know that:
Any $m$ x $n$ matrix A can be factored into $A=U\Sigma V^{T}$ , where $U$ and $V$ are orthogonal, and $\Sigma$ is of the same size as $A$ with all entries zero except down the main diagonal where the successive entries are $\sigma _{1}\geq ...\geq \sigma _{k} > 0 $ for some $k$ with $k\leq$ min$(m,n)$.
To find $U$, $\Sigma$, and $V$ , we can consider $AA^{T}$ and $A^{T}A$ which are symmetric matrices.
$AA^{T}=(U\Sigma V^{T})(V\Sigma^{T} U^{T})=U(\Sigma \Sigma^{T}) U^{T}$ ($\because$ V is orthogonal implies $V^{T}V=I_{n}$)
$A^{T}A=(V\Sigma^{T} U^{T})(U\Sigma V^{T})=V(\Sigma^{T} \Sigma)V^{T} $ ($\because$ U is orthogonal implies $U^{T}U=I_{m}$)
From Spectral Theorem, I understand that, since $AA^{T}$ and $A^{T}A$ are symmetric matrices, $U$ must be the eigenvector matrix for $AA^{T}$, and $\Sigma \Sigma^{T}$ is the eigenvalue matrix for $AA^{T}$; whereas $V$ must be the eigenvector matrix for $A^{T}A$, and $\Sigma^{T} \Sigma$ is the eigenvalue matrix for $A^{T}A$.
But, I don't understand why the $k$ singular values on the diagonal of $\Sigma$ are the square roots of the nonzero eigenvalues of both $AA^{T}$ and $A^{T}A$. It seems like this is only true if $\Sigma \Sigma^{T}$=$\Sigma ^{2}$ and $\Sigma^{T} \Sigma$=$\Sigma ^{2}$ . But $\Sigma \Sigma^{T}$ is $m$ x $m$ matrix, whereas $\Sigma^{T} \Sigma$ is $n$ x $n$ matrix. How can both of them be equal to $\Sigma^{2}$ ? 
I'm so confused. :(
 A: Let's just multiply some matrices.
\begin{align*}
\begin{pmatrix}
2 & 0 & 0 \\
0 & 3 & 0 \\
0 & 0 & 0 \\
0 & 0 & 0
\end{pmatrix}
\begin{pmatrix}
2 & 0 & 0 & 0 \\
0 & 3 & 0 & 0 \\
0 & 0 & 0 & 0
\end{pmatrix}
&=
\begin{pmatrix}
4 & 0 & 0 & 0 \\
0 & 9 & 0 & 0 \\
0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0
\end{pmatrix}, \\
\begin{pmatrix}
2 & 0 & 0 & 0 \\
0 & 3 & 0 & 0 \\
0 & 0 & 0 & 0
\end{pmatrix}
\begin{pmatrix}
2 & 0 & 0 \\
0 & 3 & 0 \\
0 & 0 & 0 \\
0 & 0 & 0
\end{pmatrix}
&=
\begin{pmatrix}
4 & 0 & 0 \\
0 & 9 & 0 \\
0 & 0 & 0 
\end{pmatrix}.
\end{align*}
See how the non-zero diagonal entries of $\Sigma\Sigma^T$ and $\Sigma^T\Sigma$ agree?
A: The $\Sigma$ matrix is a sabot matrix which insures conformability between $\mathbf{U}$ and $\mathbf{V}^{*}$. In block form, use the diagonal matrix of singular values $\mathbf{S}_{\rho \times \rho}$ where $\rho$ is the matrix rank:
$$
\Sigma =
\left(
\begin{array}{cc}
 \mathbf{S} & \mathbf{0} \\
 \mathbf{0} & \mathbf{0} \\
\end{array}
\right)_{m\times n}, \quad
%
\Sigma^{\mathrm{T}} =
\left(
\begin{array}{cc}
 \mathbf{S} & \mathbf{0} \\
 \mathbf{0} & \mathbf{0} \\
\end{array}
\right)_{n\times m}
%
\Sigma^{\dagger} =
\left(
\begin{array}{cc}
 \mathbf{S}^{-1} & \mathbf{0} \\
 \mathbf{0} & \mathbf{0} \\
\end{array}
\right)_{n\times m}
$$
For example, if the target matrix $\mathbf{A}$ has full column rank
$$
\Sigma =
\left(
\begin{array}{c}
 \mathbf{S} \\
 \mathbf{0}
\end{array}
\right)_{m\times n}, \quad
%
\Sigma^{\mathrm{T}} =
\left(
\begin{array}{cc}
 \mathbf{S} & \mathbf{0} \\
\end{array}
\right)_{n\times m}
%
\Sigma^{\dagger} =
\left(
\begin{array}{cc}
 \mathbf{S}^{-1} & \mathbf{0} \\
\end{array}
\right)_{n\times m}
$$
and $$\Sigma^{\mathrm{T}}\Sigma = \mathbf{S}^{2}.$$
For an example showing $\Sigma$ gymnastics, see 
SVD and linear least squares problem.
