3
$\begingroup$

I am not very sophisticated in my linear algebra, so please excuse any messiness in my terminology, etc.

I am attempting to reduce the dimensionality of a dataset using Singular Value Decomposition, and I am having a little trouble figuring out how this should be done. I have found a lot of material about reducing the rank of a matrix, but not reducing its dimensions.

For instance, if I decompose using SVD: $A = USV^T$, I can reduce the rank of the matrix by eliminating singular values below a certain threshold and their corresponding vectors. However, doing this returns a matrix of the same dimensions of $A$, albeit of a lower rank.

What I actually want is to be able to express all of the rows of the matrix in terms of the top principal components (so an original 100x80 matrix becomes a 100x5 matrix, for example). This way, when I calculate distance measures between rows (cosine similarity, Euclidean distance), the distances will be in this reduced dimension space.

My initial take is to multiply the original data by the singular vectors: $AV_k$. Since $V$ represents the row space of $A$, I interpret this as projecting the original data into a subspace of the first $k$ singular vectors of the SVD, which I believe is what I want.

Am I off base here? Any suggestions on how to approach this problem differently?

|cite|improve this question
$\endgroup$
  • $\begingroup$ Are you looking for a matrix $\tilde{A}$ of reduced dimension such that $\mathcal{R}(A^T) \approx \mathcal{R}(\tilde{A}^T)$ in some sense? $\endgroup$ – copper.hat Jun 26 '12 at 17:55
  • $\begingroup$ I'm not completely familiar with your notation, but I think that's right. I want to approximate the larger matrix with a matrix of reduced dimension while maintaining as much of the original variance as possible (hence the use of the SVD). $\endgroup$ – Matt Jun 26 '12 at 18:42
3
$\begingroup$

If you want to do the 100X80 to 100X5 conversion, you can just multiply U with the reduced S (after eliminating low singular values). What you will be left with is a 100X80 matrix, but the last 75 columns are 0 (provided your singular value threshold left you with only 5 values). You can just eliminate the columns of 0 and you will be left with 100X5 representation.

The above 100x5 Matrix can be multiplied with the 5X80 matrix obtained by removing the last 75 rows of V transpose, this results in the approximation of A that you are effectively using.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thank you! This is exactly what I wanted to know. $\endgroup$ – Matt Oct 1 '12 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.