I have a $m \times n$ data matrix $X$, ($m$ instances and $n$ features) on which I calculate the Covariance matrix $C$ and perform eigenvalue decomposition. so $C=W \Sigma W'$ where $W$ are the eigenvectors and $\Sigma$ are the eigenvalues arranged in diagonal matrix.
Next, I performed SVD (Singular Value Decomposition) of $X$ , so $X=U \Sigma V'$. Now I noticed a strange thing, the eigenvectors $W$ and the right singular vectors $V$ are equal (at least in magnitude), there were some differences from 4th decimal point in the values (I used Matlab), but I guess that could just be numerical error. Every odd vector in $W$ and $V$ were showing opposite signs, while every even vector of $W$ and $V$ shows same sign !!
So why does this happen ?? ie I know that EVD and SVD are somehow connected, from the formulation, it looks like SVD is generalised form of EVD but I am not sure about that. But why does $V$ vectors and $W$ vectors equal in magnitude ? what does this mean intuitively ? I have used PCA and SVD and have seen the term loading matrix used for both $W$ and $V$, so I knew that should be related, But can someone explain this ??
And what is the significance of left singular vectors ? are they useful anywhere ?
Finally, I have a question about Diagonal matrices. Wikipedia in its page on orthogonal matrices says that a matrix Q is orthogonal if its transpose is equal to its inverse: or $Q' = Q^{-1}$ or $QQ' = Q'Q = I$. But for diagonal matrices the first equation does not hold true, but the second equation holds true, (this is because the inverse of diagonal matrix is 1/main diagonal elements). So does that mean a diagonal matrix is not orthogonal ? Can someone clarify this difference ?