5
$\begingroup$

For a real symmetric matrix A, it's true that:

$$A A^T = A^T A = A^2$$

And since the right and left singular vectors of $A$ are the eigenvectors of $A^T A$ and $A A^T$ respectively, the right and left singular vectors ought to be identical.

Here's the particular situation that is causing my confusion. Let

$$A = \begin{bmatrix} 4 & 1 & -2 & 2 \\ 1 & 2 & 0 & 1 \\ -2 & 0 & 3 & -2 \\ 2 & 1 & -2 & -1\end{bmatrix}$$

Now when I try to take the SVD of this matrix, the matrices $U$ and $V$, of the left and right singular vectors respectively, are identical. Just as I thought they would be. In particular, for $U$, $\Sigma$, and $V$ I get:

$$U = V = \begin{bmatrix} -0.718 & 0.202 & 0.177 & -0.642 \\ -0.221 & 0.789 & 0.178 & 0.544 \\ 0.557 & 0.580 & -0.288 & -0.520 \\ -0.353 & 0.010 & -0.924 & 0.144 \end{bmatrix}$$

and

$$\Sigma = \begin{bmatrix} 6.845 & 0 & 0 & 0 \\ 0 & 2.269 & 0 & 0 \\ 0 & 0 & 2.198 & 0 \\ 0 & 0 & 0 & 1.084 \end{bmatrix}$$

But if I use this to calculate $U \Sigma V^T$ I don't get $A$. This already confuses me, but to make matters worse, if I take $V$ to be $U$ except with the signs changed on the third column, then it comes out correctly! I feel like I must be misunderstanding something about the SVD or Eigenvectors in general, as I thought that as long as each eigenvector had norm 1, it didn't matter what the sign of it was (i.e. I think about it as more of an eigendirection). Can anyone point out why I'm getting these results?

$\endgroup$
1
  • $\begingroup$ Have you tried $U^T\Sigma U$? $\endgroup$ Mar 29, 2018 at 15:43

3 Answers 3

5
$\begingroup$

If $A$ is symmetric then $A^TA = A^2$ and the eigenvalues of $A^2$ are the squares of the eigenvalues of $A$, and therefore the singular values are $\sigma=|\lambda|$. The eigenvectors of $A^TA=A^2$ are just the same as those of $A$; call these $v_1,v_2,\ldots,v_n$ and let $V=[v_1\,v_2\,\cdots\,v_n]$. Assume for simplicity that $A$ is non-singular. Then $u = \frac{1}{\sigma} Av$ for each $v$ and corresponding singular value and therefore $u=\frac{\lambda}{|\lambda|} v = sign(\lambda) v$. Let Q be the diagonal matrix containing the numbers $sign(\lambda)$ for each $\lambda$. Then $U=VQ$, and the singular value decomposition is $A=U\Sigma V^T = V (Q \Sigma) V^T$. So, the part $Q\Sigma$ takes care of getting the correct sign for the eigenvalues and it is what you were missing, in other words, $Q\Sigma = diag(\lambda's)$

$\endgroup$
4
$\begingroup$

This is a common mistake when trying to calculate the SVD of symmetric matrices for homework type problems, the root of the issue being the non-uniqueness of eigenvectors.

We know from the SVD of $A=U\Sigma V^T$, that $A^TA = V\Sigma^2 V^T$ (remember that $A$ is symmetric), so find the eigenvectors and eigenvalues of $B=A^TA$. This gives you $V$ and $\Sigma$ (by taking a square root of the eigenvalues), respectively.

Here is the trap. If you were to try to find $U$ by computing the eigenvectors of $AA^T$, you would end up back with $B$ and might conclude that $U=V$. That is wrong, since eigenvectors are not unique. Instead, use the SVD equation, and right multiply by $V$ to get: $AV=U\Sigma=[u_1\sigma_1 \, \ldots \, u_n\sigma_n]$. You already know $A,V$ and $\Sigma$, so normalize the $i^{th}$ column of the product $AV$ by $\sigma_i$ to get $u_i$, the $i^{th}$ column of $U$. This way you get the correct SVD of $A$.

PS: If one or more of the singular values are zero, then the above approach fails. In such cases, you will need to start with those $u_i$'s which correspond to non-zero singular values, and apply a Gram Schmidt procedure with the canonical basis to find the remaining orthonormal vectors to get $U$.

$\endgroup$
3
$\begingroup$

NOTE that while U == V, the matricies U and V are not symmetric. The definition of the SVD: A = USV' (where the prime indicates the Hermitian (conjugate transpose) of the matrix) - so this is probably your mistake - you probably forgot to apply the transpose - or your math language may have already supplied it for you.

In a lot of the languages with built-in SVD (ie Matlab, Python...) you need to be careful that the SVD function might return U,S,V' (the Hermitian of V)

for example in python you would have U,S,VH = svd(A)

This produces the same U,S,V as you have above - except that it gives the Hermitian of V

Putting it all back together... (in python 'dot' is used for matrix multiplication)

B = dot(U,dot(diag(S),VH))

yielding abs(B-A).max() = 9.992e-16, somewhere around machine epsilon

$\endgroup$
13
  • $\begingroup$ For a real matrix, such as the V in my example, the Hermitian is the same as the transpose, right? In which case V' = V^T = U^T, but calculating U Sigma U^T still does not result in the right answer. $\endgroup$
    – CRice
    Apr 7, 2016 at 23:01
  • $\begingroup$ Yep B = U*S*V' should do it (in Matlab mathworks.com/help/matlab/ref/svd.html) - are you using matlab? if so enter: U*Sigma*V' $\endgroup$ Apr 7, 2016 at 23:03
  • $\begingroup$ I'm definitely not forgetting to apply the transpose, per your answer. I AM applying the transpose, but still getting the wrong result. Using the U,S,V from my example, U S V^T = U S U^T != A. I'd also like to add that if I use an external SVD calculation, such as this one, U and V are not identical (why?) but rather V is identical to U except for the third column. $\endgroup$
    – CRice
    Apr 7, 2016 at 23:12
  • $\begingroup$ are you using the full precision of the numbers in the matrices - and the ones shown up above are just rounded off? I will have a look at the external one (when I use the full precision numbers in python - I get a fair representation of your matrix) $\endgroup$ Apr 7, 2016 at 23:17
  • $\begingroup$ I see that in the external calculator - the U and V are different - in python I am also noticing that they are not equal - it then occurs to me that U=V only in the case of a positive definite symmetric matrix (all the eigenvalues are positive) - this is not the case for your matrix [6.84462111, -2.19751698, 1.08436446, 2.26853141] ] - you need a separate U and V - if we do this same trick on the matrix: P = AA^T - then U = V $\endgroup$ Apr 7, 2016 at 23:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .