I noticed that Axler's Linear Algebra Done Right has an explanation of SVD in a matrix free way. The statement of the theorem is the following

Given $T\in L(V)$ there are orthonormal basis $(e_1,\ldots,e_n)$ and $(f_1,\ldots,f_n)$ of $V$ such that $Tv = s_1\langle v,e_1\rangle f_1+\ldots+s_n\langle v,e_n\rangle f_n$, where $s_1,\ldots,s_n$ are the singular values of $T$.

This is easy to prove by just letting $(e_1,\ldots,e_n)$ the an orthonormal basis for $T^*T$, which is guaranteed to have such a basis being self-adjoint. We then apply the polar composition by writing $T=S\sqrt{T^*T}$, where $S$ is an isometry, so it preserves orthonormality of vectors and we can set $f_i = Sf_i$ and seeing that $Tv$ has the required form is then a trivial computation.

My question is the following: In most applications of SVD, where we do some form of dimensionality reduction, we do not have a square matrix, so the assumption $T\in L(V)$ doesn't hold and we would instead need to generalize this to the setting $T\in L(V,W)$, where $V$ and $W$ are vector spaces of possible different dimensions. Is there a nice formulation of the SVD in this setting comparable to the one above?

  • 1
    $\begingroup$ Maybe the formula $\sum \sigma_ju_jv_j^*$ in page 6 of math.vt.edu/people/embree/cmda3606/chapter8.pdf is what you want ? $\endgroup$ – paf Jun 2 '17 at 16:32
  • 1
    $\begingroup$ Yes, it's the same except $f_i$ is now an orthonormal basis of $W$. $\endgroup$ – Qiaochu Yuan Jun 2 '17 at 16:52
  • $\begingroup$ I'm not sure I follow. We can clearly choose an orthonormal basis of eigenvectors for $T^*T$. However, we're summing over the length of that basis which will be different, if $W$ is lower or higher-dimensional? $\endgroup$ – eof Jun 2 '17 at 17:08
  • $\begingroup$ It depends on how you index the singular values. If you index the singular values so that some of them are zero, then the singular vectors corresponding to zero singular values are highly non-unique; you basically pick them arbitrarily so that you continue to have an orthonormal basis. If $\dim V = n$ and $\dim W = m$ there are at most $\text{min}(n, m)$ nonzero singular values (in fact there are exactly $\text{rank}(T)$ nonzero singular values), so one convention is to only index up this far. $\endgroup$ – Qiaochu Yuan Jun 3 '17 at 4:46

The cleanest way to extend the matrix-free statement of the SVD to the case of linear maps from $V$ to $W$ is as follows (note the addition of the word "nonzero" and the change from "bases" to "lists"):

Suppose the nonzero singular values of $T \in \mathcal{L}(V, W)$ are $s_1, \dots, s_n$. Then there exist orthonormal lists $e_1, \dots, e_n$ in $V$ and $f_1, \dots, f_n$ in $W$ such that $$ Tv = s_1 \langle v, e_1 \rangle f_1 + \dots + s_n \langle v, e_n \rangle f_n $$ for every $v \in V\!$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.