Apologies that this turned out a little long.
I have studied the Singular Value Decomposition in Fundamental of Matrix Computations by Watkins and I couldn't to solve the Exercise 4. 1. 16 which is a demonstration of the following theorem:
(Watkins) Theorem 4.1.1 (SVD Theorem) Let $A\in \mathbb{R}^{n \times m}$ be a nonzero matrix with rank $r$. Then $A$ can be expressed as a product \begin{equation} A = U \Sigma V^{T} \end{equation} where $U \in \mathbb{R}^{n \times n}$ and $V \in \mathbb{R}^{m \times m}$ are orthogonal, and $\Sigma \in \mathbb{R}^{n \times m}$ is a rectangular "diagonal" matrix \begin{equation} \Sigma = \left( \begin{array}{cccccc} {\sigma_{1}} & {} & {} & {} & {} & {}\\ { } & {\sigma_2} & {} & {} & {} & {}\\ {} & {} & {\ddots} & {} & {} & {}\\ {} & {} & {} & {\sigma_r} & { } & {} \\ {} & {} & {} & {} & { } & {\ddots} \end{array} \right) \;\;\; \sigma_{1}\geq \sigma_2 \geq \cdots\geq \sigma_r > 0. \end{equation}
The Exercise is:
(Watkins) Exercise 4.1.16 In this exercise you will prove Theorem 4.1.1 by induction on $r$, the rank of $A$.
(a) Suppose $A \in \mathbb{R}^{n \times m}$ has rank 1. Let $u_1 \in \mathbb{R}^{n}$ be a vector in $R(A)$ such that $|| u_1 ||_2 = 1$. Show that every column of $A$ is a multiple of $u_1$. Show that $A$ can be written in the form $ A = \sigma_1 u_1 v_{1}^{T}$, where $v_1 \in \mathbb{R}^{m}$, $|| u_1 ||_2 = 1$, and $\sigma_1 > 0$.
(b) Continuing from part (a), demonstrate that there is an orthogonal matrix $U \in \mathbb{R}^{n \times n}$ whose first column is $u_1$. (For example, $U$ can be a reflector that maps the unit vector $e_1$ to $u_1$.) Similarly there is an orthogonal $V \in \mathbb{R}^{m \times m}$ whose first column is $v_1$. Show that $A = U \Sigma V^{T}$, where $\Sigma \in \mathbb{R}^{n \times m}$ has only one nonzero entry, $\sigma_1$, in position $(1,1)$. Thus every matrix of rank 1 has an $SVD$.
(c) Now suppose $A \in \mathbb{R}^{n \times m}$ has rank $r > 1$. Let $v_1$ be a unit vector in the direction of maximum magnification by $A$, i.e., $|| v_1 ||_2 = 1$, and $|| A v_1 ||_2 = \max_{|| v ||_{2} = 1} || Av||_2$. Let $\sigma_{1}=\left\|A v_{1}\right\|_{2}=\|A\|_{2}$, and let $u_1 = \sigma_{1}^{-1}Av_1$. Let $\tilde{U} \in \mathbb{R}^{n \times n}$ and $\tilde{V} \in \mathbb{R}^{m \times m}$ be orthogonal matrices with first column $u_1$ and $v_1$, respectively. Let $\tilde{A} = \tilde{U}^{T} A \tilde{V}$, so that $A = \tilde{U} \tilde{A} \tilde{V}^{T}$. Show that $\tilde{A}$ has the form \begin{equation} \tilde{A}=\left[ \begin{array}{cc}{\sigma_{1}} & {z^{T}} \\ {0} & {\hat{A}}\end{array}\right], \end{equation} where $z \in \mathbb{R}^{m-1}$ and $\hat{A} \in \mathbb{R}^{(n-1) \times (m-1)}$.
(d) Show that in matrix $\tilde{A}$ above the vector $z$ must be zero. You may do this as follows: Let $w=\left[ \begin{array}{c}{\sigma_{1}} \\ {z}\end{array}\right] \in \mathbb{R}^{m}$. Show that $\|\tilde{A} w\|_{2} /\|w\|_{2} \geq \sqrt{\sigma_{1}^{2}+z^{T} z}$. Then show that this inequality forces $z = 0$. Thus \begin{equation} \tilde{A}=\left[ \begin{array}{cc}{\sigma_{1}} & {0} \\ {0} & {\hat{A}}\end{array}\right]. \end{equation}
(e) Show that $\hat{A}$ has rank $r-1$. By the induction hypothesis $\hat{A}$ has an SVD $\hat{A} = \hat{U} \hat{\Sigma} \hat{V}^{T}$. Let $\sigma_2 \geq \sigma_3 \geq \cdots \sigma_r$ denote the positive main-diagonal entries of $\hat{\Sigma}$. Show that $\sigma_1 \geq \sigma_2$. Embed the SVD $\hat{A} = \hat{U} \hat{\Sigma} \hat{V}^{T}$ in larger matrices to obtain an SVD of $A$. Then use the equation $A = \tilde{U} \tilde{A} \tilde{V}^{T}$ to obtain an SVD of $A$.
Question:
I know the exercise is well done (it's a step by step that I should probably complete the demonstration easily). But I could not do it and it also did not help me what I read in the book of Strang and Golub that I took to help me solve this exercise. So, I'd like to understand the solutions this Theorem/Exercise.
