# How does the left singular value decomposition change when one duplicates a column in the original matrix?

Let $\mathbf{v}_1,\ldots,\mathbf{v}_n$ vectors in an $m$-dimensional space $V$. Taking these as column vectors of the matrix $M$, let $$M = U\Sigma V^\ast$$ its singular value decomposition. Now, I have a problem where $n>m$ and some of the $\mathbf{v}_i$ are equal*. Is there an easy (meaning, computionally efficient – it is a numerical application) way to calculate $U\Sigma V^\ast$ from $U'\Sigma' V'^\ast$, the SVD of $M$ with duplicate columns removed, or will I just need to brute-force it? Specifically, I only need the left SVD, i.e. $U$ and $\Sigma$. It seems to my like this should be possible, but I have no idea how to approach it.

*In fact, I envision cases where most of them are duplicates, which is why I'm concerned about the performance penalty of calculating the full decomposition of $M$.

• What's [svd] and why do we need a whole new tag for it? Nov 25, 2012 at 17:57
• @AsafKaragila not sure if we need this tag, I did submit a tag wiki entry. It's IMO comparable to eigenvalues in significance. Nov 25, 2012 at 17:58

I don't think there is any simple relationship between the SVD of $M$ and the SVD of $M'$, but if you keep inserting new columns into $M$ and want to compute $U$ and $\Sigma$ after each insertion, and if the number of columns of $M$ far exceeds $m$, then what you need perhaps is rank-1 update of symmetric eigenvalue decomposition of $MM^\ast$.
You see, if $MM^\ast = U\Lambda U^\ast$ is an eigenvalue decomposition of $MM^\ast$, the columns of $U$ and the diagonal of $\sqrt{\Lambda}$ are respectively the left singular vectors and the singular values of $M$. So, if right singular vectors are not needed, then a SVD problem can be converted into a symmetric eigenvalue decomposition problem. Now, if you add a new column $x$ to $M_{\textrm{old}}$ to get a new matrix $M_{\textrm{new}}$, then $M_{\textrm{new}}M_{\textrm{new}}^\ast = M_{\textrm{old}}M_{\textrm{old}}^\ast + xx^\ast$. So, given an eigenvalue decompositon of $M_{\textrm{old}}M_{\textrm{old}}^\ast$, you are going to find a new eigenvalue decompositon when $M_{\textrm{old}}M_{\textrm{old}}^\ast$ is incremented by a rank-1 matrix $xx^\ast$.
Since adding columns doesn't change the rank in your case, the left singular vectors $U$ remains the same. But the same cannot be said about singular values and the right singular vectors. To see this, look at the frobenius norm of the new matrix, it changes, so the singular values must change (since frobenius norm is also equal to sum of squares of singular values). Also, when you add a new column, the dimension of row-space increases (geometricallY). So you can use only $U$ obtained from the previous computation before adding duplicate columns. To find the singular values, call the new matrix $A_1$ (after adding duplicate columns). Now find the square roots of eigenvalues of $A_1^{T}A_1$. This will give you the singular values. I don't think (in general) you can find the new singular values using previous decomposition, other than finding $U$.
• I don't understand. Unless the column vectors of $M$ are the singular vectors themselves, why would the left singular vectors remain the same? Nov 26, 2012 at 9:53
• Isn't the left singular vectors, an orthogonal basis for the columns of the matrix. In his case, the dimension of column space nor its basis is changing. So shouldn't $U$ be fixed?. Nov 26, 2012 at 13:25
• Left singular vectors are basis vectors of (a superspace of) the column space, but the converse is not true. Adding a column to $M$ can change its left singular vectors if the right singular vectors $V$ do not form a permutation matrix. You may try some small examples. For instance, $M'=\begin{pmatrix}1&1\\0&1\end{pmatrix}$ and $M=\begin{pmatrix}1&1&1\\0&1&1\end{pmatrix}$. In both cases, both $\Sigma'$ and $\Sigma$ have distinct singular values, so $U'$ and $U$ are uniquely determined up to multiples of unit diagonal matrices. But $U'$ and $U$ are quite different. Nov 26, 2012 at 14:24