When decomposing a matrix by SVD why does $\text{Span}(Av_i)=\text{Span(col A)}$? This is from Lay, Linear Algebra 7.4 pg 417 supposedly proving that $\text{Span}(Av_i)=\text{Span(col A)}$.
I understand that each eigenvector is orthogonal, but the part of the proof that $Av_i$'s span Col A seems circular. From how I understand it, the y in the column space is in the span of the $Av_i$'s because we wrote it in a form where it looks like it's in the span. The $Av_i$'s of this not square matrix don't even necessarily  have the same length (they are the length of row dimension of A) of column space. Where is it proven that the col space A is dimension r and where is it proven that $Av_i$ spans that? Why is $Av_i$ in the col space and why does rank($A^TA$) have the same dimension r as col A? 
 A: The easiest to answer of your questions is: why is $A\mathbf{v}_i$ in the col-space of $A$? The answer is: any vector of the form $A\mathbf{x}$ is in the col-space of $A$, you could take this as the definition of the col-space. Alternatively: if we view the col-space as the set of linear combinations of the columns of $A$, let $\mathbf{a}_1, \ldots, \mathbf{a}_p$ be the columns of $A$ and let $\mathbf{x} = \begin{pmatrix} x_1 \\ \vdots \\ x_p \end{pmatrix}$. Then $A\mathbb{x} = x_1 \mathbf{a}_1 + x_2 \mathbf{a}_2 + \ldots + x_p \mathbf{a}_p$ so it is indeed in $Col A$.
Secondly: the book also uses the above in the opposite direction, when it says that $\mathbf{y}$ is $A\mathbf{x}$ for some $\mathbf{x}$ (of length $n$). They can say that because we were looking at not arbitrary $\mathbf{y}$ but at $\mathbf{y}$ in the column space of $A$. The (hidden) result they use is that EVERY $\mathbf{y}$ in $Col(A)$ is of the form $\mathbf{y} = A\mathbf{x}$. This follows easily from either of the two definitions of col-space I gave above. If $\mathbf{x}$ would not exist, then $\mathbf{y}$ were not in the column space, so since it is given that $\mathbf{y}$ does live in Col(A), we can assume existence of $\mathbf{x}$. 
This explains the 'circular' part: we know that $\mathbf{x}$ can be written as linear combination of the $\mathbf{v}_i$ because of the nice properties of the square matrix $A^TA$ (which is much nicer than $A$ itself) and only after having established that fact, we can multiply both sides of the equation $\mathbf{x} = c_1\mathbf{v}_1 + \ldots + c_n\mathbf{v}_n$ by $A$ to obtain: $A\mathbf{x} = c_1 A \mathbf{v}_1 + \ldots + c_n A \mathbf{v}_n$. This latter equation is in the book, only it writes $\mathbf{y}$ for $A\mathbf{x}$.
So we now know that all $n > r$ of the $A\mathbf{v}_i$ together span $Col(A)$ and we need to do some extra work to see that actually only the first $r$ of them are enough. I leave this to you for now, and maybe come back to it later.
Good luck!
