5
$\begingroup$

For a real symmetric matrix A, it's true that:

$$A A^T = A^T A = A^2$$

And since the right and left singular vectors of $A$ are the eigenvectors of $A^T A$ and $A A^T$ respectively, the right and left singular vectors ought to be identical.

Here's the particular situation that is causing my confusion. Let

$$A = \begin{bmatrix} 4 & 1 & -2 & 2 \\ 1 & 2 & 0 & 1 \\ -2 & 0 & 3 & -2 \\ 2 & 1 & -2 & -1\end{bmatrix}$$

Now when I try to take the SVD of this matrix, the matrices $U$ and $V$, of the left and right singular vectors respectively, are identical. Just as I thought they would be. In particular, for $U$, $\Sigma$, and $V$ I get:

$$U = V = \begin{bmatrix} -0.718 & 0.202 & 0.177 & -0.642 \\ -0.221 & 0.789 & 0.178 & 0.544 \\ 0.557 & 0.580 & -0.288 & -0.520 \\ -0.353 & 0.010 & -0.924 & 0.144 \end{bmatrix}$$

and

$$\Sigma = \begin{bmatrix} 6.845 & 0 & 0 & 0 \\ 0 & 2.269 & 0 & 0 \\ 0 & 0 & 2.198 & 0 \\ 0 & 0 & 0 & 1.084 \end{bmatrix}$$

But if I use this to calculate $U \Sigma V^T$ I don't get $A$. This already confuses me, but to make matters worse, if I take $V$ to be $U$ except with the signs changed on the third column, then it comes out correctly! I feel like I must be misunderstanding something about the SVD or Eigenvectors in general, as I thought that as long as each eigenvector had norm 1, it didn't matter what the sign of it was (i.e. I think about it as more of an eigendirection). Can anyone point out why I'm getting these results?

$\endgroup$
  • $\begingroup$ Have you tried $U^T\Sigma U$? $\endgroup$ – Ernesto Iglesias Mar 29 '18 at 15:43
2
$\begingroup$

NOTE that while U == V, the matricies U and V are not symmetric. The definition of the SVD: A = USV' (where the prime indicates the Hermitian (conjugate transpose) of the matrix) - so this is probably your mistake - you probably forgot to apply the transpose - or your math language may have already supplied it for you.

In a lot of the languages with built-in SVD (ie Matlab, Python...) you need to be careful that the SVD function might return U,S,V' (the Hermitian of V)

for example in python you would have U,S,VH = svd(A)

This produces the same U,S,V as you have above - except that it gives the Hermitian of V

Putting it all back together... (in python 'dot' is used for matrix multiplication)

B = dot(U,dot(diag(S),VH))

yielding abs(B-A).max() = 9.992e-16, somewhere around machine epsilon

$\endgroup$
  • $\begingroup$ For a real matrix, such as the V in my example, the Hermitian is the same as the transpose, right? In which case V' = V^T = U^T, but calculating U Sigma U^T still does not result in the right answer. $\endgroup$ – CRice Apr 7 '16 at 23:01
  • $\begingroup$ Yep B = U*S*V' should do it (in Matlab mathworks.com/help/matlab/ref/svd.html) - are you using matlab? if so enter: U*Sigma*V' $\endgroup$ – eigenjohnson Apr 7 '16 at 23:03
  • $\begingroup$ I'm definitely not forgetting to apply the transpose, per your answer. I AM applying the transpose, but still getting the wrong result. Using the U,S,V from my example, U S V^T = U S U^T != A. I'd also like to add that if I use an external SVD calculation, such as this one, U and V are not identical (why?) but rather V is identical to U except for the third column. $\endgroup$ – CRice Apr 7 '16 at 23:12
  • $\begingroup$ are you using the full precision of the numbers in the matrices - and the ones shown up above are just rounded off? I will have a look at the external one (when I use the full precision numbers in python - I get a fair representation of your matrix) $\endgroup$ – eigenjohnson Apr 7 '16 at 23:17
  • $\begingroup$ I see that in the external calculator - the U and V are different - in python I am also noticing that they are not equal - it then occurs to me that U=V only in the case of a positive definite symmetric matrix (all the eigenvalues are positive) - this is not the case for your matrix [6.84462111, -2.19751698, 1.08436446, 2.26853141] ] - you need a separate U and V - if we do this same trick on the matrix: P = AA^T - then U = V $\endgroup$ – eigenjohnson Apr 7 '16 at 23:33
3
$\begingroup$

If $A$ is symmetric then $A^TA = A^2$ and the eigenvalues of $A^2$ are the squares of the eigenvalues of $A$, and therefore the singular values are $\sigma=|\lambda|$. The eigenvectors of $A^TA=A^2$ are just the same as those of $A$; call these $v_1,v_2,\ldots,v_n$ and let $V=[v_1\,v_2\,\cdots\,v_n]$. Assume for simplicity that $A$ is non-singular. Then $u = \frac{1}{\sigma} Av$ for each $v$ and corresponding singular value and therefore $u=\frac{\lambda}{|\lambda|} v = sign(\lambda) v$. Let Q be the diagonal matrix containing the numbers $sign(\lambda)$ for each $\lambda$. Then $U=VQ$, and the singular value decomposition is $A=U\Sigma V^T = V (Q \Sigma) V^T$. So, the part $Q\Sigma$ takes care of getting the correct sign for the eigenvalues and it is what you were missing, in other words, $Q\Sigma = diag(\lambda's)$

$\endgroup$
2
$\begingroup$

This is a common mistake when trying to calculate the SVD of symmetric matrices for homework type problems, the root of the issue being the non-uniqueness of eigenvectors.

We know from the SVD of $A=U\Sigma V^T$, that $A^TA = V\Sigma^2 V^T$ (remember that $A$ is symmetric), so find the eigenvectors and eigenvalues of $B=A^TA$. This gives you $V$ and $\Sigma$ (by taking a square root of the eigenvalues), respectively.

Here is the trap. If you were to try to find $U$ by computing the eigenvectors of $AA^T$, you would end up back with $B$ and might conclude that $U=V$. That is wrong, since eigenvectors are not unique. Instead, use the SVD equation, and right multiply by $V$ to get: $AV=U\Sigma=[u_1\sigma_1 \, \ldots \, u_n\sigma_n]$. You already know $A,V$ and $\Sigma$, so normalize the $i^{th}$ column of the product $AV$ by $\sigma_i$ to get $u_i$, the $i^{th}$ column of $U$. This way you get the correct SVD of $A$.

PS: If one or more of the singular values are zero, then the above approach fails. In such cases, you will need to start with those $u_i$'s which correspond to non-zero singular values, and apply a Gram Schmidt procedure with the canonical basis to find the remaining orthonormal vectors to get $U$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.