4
$\begingroup$

Attempting to solve the SVD of this matrix: \begin{bmatrix} 2 & 0 \\ 0 & -3 \\ 0 & 0 \end{bmatrix}

I know that one possible (my understanding is the SVD is not necessarily a unique result) result is: $$ \begin{bmatrix} 1 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \cdot \begin{bmatrix} 2 & 0 \\ 0 & 3 \\ 0 & 0 \end{bmatrix} \cdot \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}^T = \begin{bmatrix} 2 & 0 \\ 0 & -3 \\ 0 & 0 \end{bmatrix} $$

Which I have verified. I am trying to solve this problem myself, and am getting a different result; in particular, one that does not satisfy $A=U\Sigma V^T$. Therefore, I am trying to understand the hole in my approach.

Currently, my result looks like this: $$ \begin{bmatrix} 0 & 0 & 1 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix} \cdot \begin{bmatrix} 3 & 0 \\ 0 & 2 \\ 0 & 0 \end{bmatrix} \cdot \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}^T = \begin{bmatrix} 0 & 0 \\ 0 & 3 \\ 2 & 0 \end{bmatrix} $$

This is my approach, given a matrix $M$

1. Construct $MM^T$ and $M^TM$ $$ MM^T = \begin{bmatrix} 4 & 0 & 0 \\ 0 & 9 & 0 \\ 0 & 0 & 0 \end{bmatrix} \quad\&\quad M^TM = \begin{bmatrix} 4 & 0 \\ 0 & 9 \\ \end{bmatrix} $$

2. Solve for the eigenvalues and eigenvectors of both $MM^T$ and $M^TM$

This is accomplished via QR Factorization. Eigenvectors are listed in order of highest magnitude eigenvalue to lowest magnitude eigenvalue. The results are as follows:

$$ MM^T \enspace eigenvalues = \begin{bmatrix} 9 & 4 & 0 \end{bmatrix} $$ $$ MM^T \enspace eigenvectors = \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix} \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix} \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} $$ $$ M^TM \enspace eigenvalues = \begin{bmatrix} 9 & 4 \end{bmatrix} $$ $$ M^TM \enspace eigenvectors = \begin{bmatrix} 0 \\ 1 \end{bmatrix} \begin{bmatrix} 1 \\ 0 \end{bmatrix} $$

3. Construct $U$ using the eigenvalue-ordered eigenvectors of $MM^T$ as the columns of the matrix. Construct $V$ in the same way, but with $M^TM$.

I would also apply Gram-Schmidt orthonormalization here, but these matrices are already orthonormal.

$$ U = \begin{bmatrix} 0 & 0 & 1 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix} $$ $$ V = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} $$

4. Finally, construct $\Sigma$ by making a diagonal matrix whose values are the square roots of the non-zero eigenvalues from $MM^T$ or $M^TM$.

Presumably, I should preserve the eigenvalue order when I do this?

$$ \Sigma = \begin{bmatrix} 3 & 0 \\ 0 & 2 \\ 0 & 0 \end{bmatrix} $$

So my question is, if anyone can aid me in identifying either the mistake I've made, or the hole in my approach to this problem. Assuming that I haven't made a simple error, I believe I am missing some sort of insight as to the construction of these matrices (because you can see, for instance, that the known solution has very similar values in differing orders and signs).

$\endgroup$
  • $\begingroup$ I don't think you can expect to find the eigenvectors of $M^TM$ and $MM^T$ separately. The second set is constructed from the first. See math.stackexchange.com/questions/2276374/… $\endgroup$ – Matthew Leingang Aug 17 '18 at 19:09
  • $\begingroup$ @MatthewLeingang I'm not sure how to apply it yet, but that seems like the kind of insight I was missing. I'll have to read your other answer. $\endgroup$ – Nick Dechiara Aug 17 '18 at 19:14
  • $\begingroup$ Also, I think your $MM^T$ eigenvectors and eigenvalues are not in corresponding order. You may have swapped the eigenvectors corresponding to $0$ and $4$. The minus sign error makes sense because you both $MM^T$ and $M^TM$ are a kind of squaring. $\endgroup$ – Matthew Leingang Aug 17 '18 at 19:39
  • $\begingroup$ @NickDechiara you might also find my answer here to be useful $\endgroup$ – Omnomnomnom Aug 17 '18 at 22:01
1
$\begingroup$

First, you slightly messed-up the order of the eigenvectors. For $MM^T$ is $u_1 =(0,1,0)$, $u_2 = c(1,0,0)$ and $u_3 = c(0,0,1)$ correspond to $9$, $4$ and $0$. For $M^T M$ the result is immediate with $v_1 = c(0,1)$ and $v_2 = (-1,0)$ correspond to $9$ and $4$.

Second, some more comments: It seems like you over-killed with the computations. Note that $MM^T$ is a diagonal matrix, hence its eigenvalues are the diagonal entries with orthogonal (normalized) eigenvectors. The non-zero eigenvalues of $M^TM$ are the same, while the eigenvectors (due to symmetry) are also the same up-to a sign. Namely, once you see $MM^T$ everything else is follows immediately without Gram-Schimdt, QR and so on. Regarding the choice of the signs - it is slightly more subtle, as the singular vectors are uniquely determined only up to a sign. As such you have to be careful with the sign choice in order to "reconstruct" the original $M$. Please see here Calculating SVD by hand: resolving sign ambiguities in the range vectors. the last answer about the choice of a sign.

$\endgroup$
  • $\begingroup$ Thank you, this set me on the right path; I found the issue with my eigenvector calculations and also managed to calculate V from U instead of calculating them separately. I still need to figure out how to assign the signs (I understand that the signs of the vectors in V and U are connected, but determining how to choose those in order to recreate M is eluding me; I will keep reading). $\endgroup$ – Nick Dechiara Aug 21 '18 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.