Help with Singular Value Decomposition

I have a matrix $A$, $n\times m$. With singular value decomposition we have $A=U\Sigma V^{T}$

Why the singular values of matrix $\Sigma$ are in descending order?

• It should actually be "non-increasing"; a matrix can have repeated singular values. – J. M. is not a mathematician Dec 28 '16 at 15:56

People like uniqueness in their decompositions. There are usually some ambiguities in them, but we try to reduce them as far as posible. For the SVD, we know that the singular values are always real-valued (in fact also nonnegative) which allows us to order them. If they are distinct, this eliminates permutation ambiguities among the singular vectors/values.

For the eigenvalue decomposition we cannot do this, since the eigenvalues are in general complex which forbids a unique ordering.

The SVD is still not fully unique though, you can for instance absorb a minus sign into a column of $U$ and the matching columns of $V$ without changing the result. Not much we can do about that.

If you permute the columns of $U$ and apply the same permutation to the columns of $V$ (or rows of $V^t$), and permute the corresponding entries of $\Sigma$, you'll get an equally good decomposition with the singular values NOT in decreasing order. But the usual definition requires decreasing order as a way to make the SVD unique, at least for matrices with distinct singular values.

By the way, when there are repeated singular values, there will be many possible SVDs. For instance, if $A = I$, then $A = Q I Q^t$ for any orthgonal matrix $Q$.

By the way, even with decreasing singular values, you don't get true uniqueness: If you negate any row of $U$ and the corresponding column of $V$, the decomposition remains correct.

To be clear about my first claim:

Let $q = \sqrt{2}/2, c = \frac{1}{2} = \cos{\frac{\pi}{3}}, S = \frac{\sqrt{3}}{2} = \sin{\frac{\pi}{3}}$. Then for the matrix $$M = \begin{bmatrix} -q && -q \\ -q && q \end{bmatrix} \begin{bmatrix} 2q && 0 \\ 0 && q \end{bmatrix} \begin{bmatrix} -c && s \\ s && c \end{bmatrix},$$ we also have $$M = \begin{bmatrix} -q && -q \\ q && -q \end{bmatrix} \begin{bmatrix} q && 0 \\ 0 && 2q \end{bmatrix} \begin{bmatrix} s && c \\ -c && s \end{bmatrix},$$

So if you allow yourself to permute the singular values, then the decomposition can change.

If you fix the order of the singular values, even if this order is not decreasing, and if all singular values are distinct, then the SVD is unique up to negating any column of $U$ and the corresponding column of $V$ (or row of $V^t$).

• What do you mean to make the SVD unique? Do you mean to be one unique solution for $U, \Sigma, V$? – user105627 Dec 28 '16 at 16:17
• Yes: in general, we like to say that $U, \Sigma, V$ is the SVD rather than a SVD. If you don't specify an order, then my "permutation trick" lets you change one SVD into another; once you specify an order, you've almost got uniqueness. (See post-comment addition.) – John Hughes Dec 28 '16 at 16:19
• If I understood well, if the singular values are not decreasing then this means that the SVD is not unique? If so, I don't see why though – user105627 Dec 28 '16 at 16:21
• See my further comments (and I've corrected a swapped "row" and "column" in the earlier answer). – John Hughes Dec 28 '16 at 17:09
• Thanks a lot John. Now make sense. – user105627 Dec 29 '16 at 13:55