Interpretation of SVD for non-square matrices. I was reading the Wikipedia article on Singular Value Decomposition. It shows a nice visualisation where the SVD of a matrix $M = U\Sigma V^*$ allows us interpret M as a rotation $V^*$, followed by a scaling $\Sigma$, and a second and final rotation $U$. That makes perfect sense when $M$ is a square.
Now, how can I interpret the decomposition of a non-square matrix? If I take as a simple example a real row matrix $M$, then $U$ always seems to be a scalar equal to 1 or -1 (which also makes total sense), and $\Sigma$ has a single non-zero entry, namely the first SV. And what does the matrix $V^*$ represent?
 A: First we will assume that $M$ is a real matrix. In the complex case, the geometry of the SVD is a little harder to visualize because taking inner products also requires conjugation. 
Also, let us define the SVD as follows: if $M$ is $m \times n$, then the SVD is given by
$$
M = U \Sigma V^*,
$$
where $U,V$ have orthonormal columns, $\Sigma$ is diagonal with non-negative entries, and $U$ is $m \times r$, $\Sigma$ is $r \times r$, and $V^*$ is $r \times n$, and $r = \mathrm{rank}(M)$. I believe this definition of SVD is often called the "thin" or "reduced" version.
With these assumption, it's best to make sense of $U$ and $V$ as storing orthonormal bases for the range and corange (also called column space and row space when referring to matrices), respectively, when $M$ is thought of as a linear map from $\mathbb{R}^n$ to $\mathbb{R}^m$.
So the correct picture would be, first take the $n$-dimensional ball. Then look at its image under $M$; this will be an $m$-dimensional ellipsoid (and its interior). The lengths of the axes of this resulting ellipsoid are the singular values, the normalized directions of these axes are the columns of $U$, and the preimages of those axes are the columns of $V$.
For $n = m$, this corresponds to some rotation and stretching since we're mapping an $n$-sphere to an $n$-ellipsoid.
For your example where $M$ is just a row matrix, then the row matrix $V^{\ast}$ is just the normalized version of $M$.
A: As you are asking for a specific understanding of the meaning of th SVD for a non-square matrix, I propose you to have a look at the following method, which gives a natural interpretation to the dominant triple of matrix A.
It is (see diagram below) an iterative  method for the computation of the triple $U_1,V_1,\sigma_1$ where $\sigma_1$ is the largest singular value, and $U_1$, resp. $V_1$ are the first columns of $U$ and $V$ in the SVD decomposition $A=USV^T$.
This method builds two sequences $X^{(k)}$ and $Y^{(k)}$ "converging" resp. to $X_0$ and $Y_0$. In fact, as presented, it is not the vectors that converge but their directions (said otherwise, the normalized versions $X^{(k)}/\|X^{(k)}\|$ and $Y^{(k)}/\|Y^{(k)}\|$ converge).   
Using a method called deflation, one can use (almost) the same method for computing the second, third, etc. triple. In fact there are much more efficient (and accurate) methods ; this one has the advantage of being more "natural".

