2
$\begingroup$

So I understand the proofs behind Singular Value Decomposition but I'm having trouble interpreting it in the context of a real world problem.

Specifically, If I'm given an $m\times n$ data matrix A, where we have m training examples and n features collected for each example, I'm having trouble understanding the meaning behind Av$_j$ = $\sigma$$_j$u$_j$ where L$_A$ (left multiplication by A) is our linear transformation and $\beta$ = {v$_1$, v$_2$, ... , v$_n$} is an orthonormal basis for F$^n$ and $\gamma$ = {u$_1$, u$_2$, ... , u$_m$} is an orthonormal basis for F$^m$.

From reading various posts and articles, the idea seems to be a larger $\sigma$$_j$ indicates more variation in the data along that vector u$_j$ while a smaller variation in a certain direction u$_j$ is captured with a smaller $\sigma$$_j$.

However, when we are looking at Av$_j$ = $\sigma$$_j$u$_j$ i'm not sure why we care about what L$_A$ is doing. After all, this relationship would be great if I wanted to see what L$_A$ does when it acts on a orthonormal basis $\beta$ but A is just a data matrix so i'm not sure how to interpret the range of a data matrix or what types of transformations A is making when presented a vector x to 'do' left multiplication on.

$\endgroup$
  • $\begingroup$ If you use a matrix just to store some data and not multiply it to anything, then SVD would be a useless concept $\endgroup$ – polfosol Jan 6 '17 at 10:42
2
$\begingroup$

Your data sets are the rows of $A$. Thus you get the $k$th sample by computing $e_k^TA$. Using the SVD this is also $$ e_k^TA=\sum_{j=1}^nσ_j(e_k^T{\bf u}_j)\,{\bf v}_j^T $$ You can reduce this sum to the leading $d$ terms with the largest $d$ singular values to get a good approximation of the data, which means that your data vectors are all close to the subspace spanned by ${\bf v}_1,…,{\bf v}_d$ for some suitably chosen $d<n$.

$\endgroup$
  • $\begingroup$ Thank you for the response. Would you be able to explain or derive that equation a little further? I'm not quite seeing the equality $\endgroup$ – H_1317 Jan 6 '17 at 18:12
  • 1
    $\begingroup$ The SVD is $A=U\Sigma V^T=\sum_jσ_j u_jv_j^T$. Now left-multiply with a canonical basis vector to extract the corresponding row... $\endgroup$ – Dr. Lutz Lehmann Jan 6 '17 at 18:41
  • $\begingroup$ thank you! haven't seen that matrix multiplication representation often. So at each iteration through j = 1 to n we are adding another term of the dot product for each space of the final mxn matrix, A. though here we restrict it to an individual row in the final matrix. so from what i've read if someone were to employ SVD they would set the smallest singular values to 0 then recompute the matrix, resulting in hopefully a lower rank matrix. But the resulting row here will still be the same size just with smaller values since some of the later sums in the dot product have been scaled by 0. $\endgroup$ – H_1317 Jan 6 '17 at 20:59
  • $\begingroup$ how would the smaller values result in a lower rank? $\endgroup$ – H_1317 Jan 6 '17 at 20:59
  • 1
    $\begingroup$ Yes, that is correct. You can also define the rank as the lowest number of dyadic products in a sum that equals the matrix. Thus the rank of the reduced SVD is $d$. - It is not guaranteed that the entries of the reduced matrix are smaller than the entries of the original matrix. $\endgroup$ – Dr. Lutz Lehmann Jan 8 '17 at 12:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.