1
$\begingroup$

I have a $m \times n$ data matrix $X$, ($m$ instances and $n$ features) on which I calculate the Covariance matrix

$$C := \frac{1}{(m-1)} X'X.$$

Then I perform eigenvalue decomposition of $C$, and store the eigenvalues $E$.

In another application I also require the singular values of $X$, so I perform SVD on $X$, and store the singular values $S$. Now I want to know how the eigenvalues of the covariance matrix $C$ are related to the singular values of the data matrix $X$.

Wikipedia in its SVD article says:

The non-zero singular values of $X$ (found on the diagonal entries of $\Sigma$) are the square roots of the non-zero eigenvalues of both $X'X$ and $XX'$.

However, there is no mention of the connection between the non-zero singular values of $X$ and the eigenvalues of $C$.

I tried dividing the squares of the singular values of $X$ by $m-1$, but then there is a huge difference between $\frac{S_1^2}{m-1}$ and $E_1$, where $S_1$ refers to first sigular value of $X$, and $E_1$ is the first eigenvalue of $C$. The remaining values match.

Can anyone point out to why this is happening? I used rand function in Matlab to create $X$, so the elements of $X$ are pseudorandom numbers in $(0,1)$ drawn from the uniform distribution.

$\endgroup$
1
$\begingroup$

Let singular values of $X$ be $\sigma_1 \ge \sigma 2 \ge \sigma_r > \sigma_{r+1} = \cdots = 0$, and let the eigenvalues of $C$ be $\lambda_1 \ge \lambda_2 \ge \cdots \ge \lambda_n$. Then, indeed,

$$\lambda_k = \frac{\sigma_k^2}{m-1}.$$

To see this, let $X = U \Sigma V^T$, where $U$ and $V$ are orthogonal, and $\Sigma = \mathop{\rm diag}(\sigma_1,\sigma_2,\dots)$. Then

$$C = \frac{1}{m-1} X^T X = V \left( \frac{1}{m-1} \Sigma^T \Sigma \right) V^T.\tag{1}$$

Since $\Sigma$ is rectangular diagonal, and $(1)$ is a similarity relation,

$$\frac{1}{m-1} \Sigma^T \Sigma = \mathop{\rm diag} \left( \frac{\sigma_1^2}{m-1}, \frac{\sigma_2^2}{m-1}, \dots \right) = \mathop{\rm diag} ( \lambda_1, \lambda_2, \dots).$$

Now, the important thing here is that this is theory. In practice, matrix multiplication is not numerically stable and the errors may occur. Furthermore, they may be very significant in terms of the eigenvalues of $X^TX$. This is one of the main purposes of SVD: accurate computation of the eigenvalues of $X^TX$ (without actually computing the product $X^TX$).

However, there may also be an error in your code, since these errors usually happen with the smallest eigenvalues.

If this happens only sometimes, give an example of one such $X$ (of small order, please) along with your eigenvalues of $C$ and singular values of $X$, so we can check what is going on. If it happens all the time, give us your code, as it may have a bug.

$\endgroup$
1
$\begingroup$

I was getting such an error because the input matrix to SVD needs to be mean centered. This is because when we do the eigenvalue decomposition on the Covariance matrix $C$, the data input to $C$ is mean centered.

So the right way is to mean center the data and then apply SVD, then take the square of singular values and divide by $m-1$. Then $E = \frac{S^2} {m -1 }$. holds perfectly.

So I will consider this question as solved.!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.