1
$\begingroup$

I've been on a couple Wikipedia pages today reading up on the SVD and the use of low rank approximation, and I have a couple of basic questions:

if

$$A = U\Sigma V^*$$

$$= [U_1 U_2] \begin{bmatrix} \Sigma_1 & 0 \\ 0 & \Sigma_2 \\ \end{bmatrix}[V_1 V_2]$$

then $A'=U_1\Sigma_1V_1$, called a "reduced SVD", is a rank $r$ matrix such that the Frobenius norm $||A-A'||_F$ is minimized.

So, does this mean that for some large data matrix -- let's say all the columns of $A$ represent lung cancer patients, and the rows represent variables such as the patients' age, height, weight, marital status, smoker / non-smoker, has or doesn't have family history of cancer, etc. -- with the lower rank $r$ matrix, we essentially "delete" all of the rows that are insignificant, in the sense that those rows of variables showed no variance and so isn't helpful? E.g. maybe the vast majority of patients are married, and so we delete the row corresponding to marital status. And so we keep all the rows of the matrix that have the most variance.

Intuitively, this seems wrong: based on the above, I could wrongly throw out the row variable of smoker status, if the vast majority of the patients were smokers and so there is little variance. But that would be throwing out pretty essential data that shows that most lung cancer patients were smokers.

So, where have I gone wrong in my thinking of low-rank approximation / the SVD?

Also, concerning the data matrix $A$: does it ever act on vectors via ? That would seem silly...what would its "action" even be? It's just an enormous array of the patients' data. It's not some...rotation...or dilation...or reflection....or projection...

whereas, in contrast, a stochastic matrix acting on probability vectors has the effect of 'updating' the probability vector of some Markov chain.

Thanks,

|cite|improve this question
$\endgroup$
1
$\begingroup$

If, for example, you want to predict lung cancer then your data set should have samples from both categories (i.e. healthy and sick people). If the smoking attribute was strongly associated with only sick people then it would not appear in all the data points (it would only appear for sick people) and would have a high variance.

Also remember that when you do low rank approximation you basically remove the contribution of the singular vectors that correspond to the smallest singular values. These do not necessarily correspond to the rows of the matrix $A$ but could correspond to some linear combination of them. I hope this helps.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Hi @MrHat, this was very helpful - thanks so much. I computed some SVDs by hand and watched videos of PCA last night, after posting this question. I have a curious follow-up question, if you don't mind. So people who use PCA in their work, does the software, e.g., Matlab, do everything for them? For instance, it takes awhile to compute the SVD of a toy problem 2x2 matrix by hand and not make any errors, but Matlab will give the SVD within seconds. Does it also answer all the other qualitative questions that one needs for his research, or is that the work of the individual researcher? $\endgroup$ – User001 Oct 3 '16 at 3:49
  • $\begingroup$ What does a researcher who uses PCA do, besides applying analyses from software on the data sets and getting the answers directly from these software packages? Thanks @MrHat $\endgroup$ – User001 Oct 3 '16 at 3:52
  • 1
    $\begingroup$ Glad I could help. As far as I know Matlab doesn't do any qualitative stuff on its own. Matlab will perform SVD on the Matrix as you say and will give the researcher 3 matrices. The $U$,$V$ matrices contain singular vectors and the $\Sigma$ matrix will have the singular values. The researcher can then decide how many principal components he wants to use. He can keep the singular vectors (these correspond to the directions of maximum variance as we have pointed out). He may then project onto the space of the principal components to get a new feature description $\endgroup$ – MrHat Oct 3 '16 at 13:57
  • 1
    $\begingroup$ By doing this the researcher has reduced the dimensionality of the feature space without losing really valuable information. This post gives an intuitive explanation of PCA stats.stackexchange.com/questions/2691/… $\endgroup$ – MrHat Oct 3 '16 at 14:04
  • 1
    $\begingroup$ Here is also a nice illustration onlinecourses.science.psu.edu/stat857/sites/… $\endgroup$ – MrHat Oct 3 '16 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.