# SVD and PCA - help

I have some data on cars: i have been given the Noise,size,speed,if its electric or not, if its a lorry or not.

it looks something like this:

$$\begin{matrix} Noise & Size & Speed & electric/not & lorry/not \\ 80 & 10 & 30 & 1 & 0 \\ 70 & 5 & 80 & 0 & 1 \\ ... & ... & ... & ... & ... \\ \end{matrix}$$

after this i subtracted the mean from every column and then did the SVD in MATLAB

now i am looking at VS' matrix since it asks me to analyse the principle components using V and S to find any relationships between the data.

i am assuming in the VS' matrix that the rows are the variables and the columns are the principle components something like this:

$$\begin{matrix} variable & PC1 & PC2 & PC3 & PC4 \\ Noise & -4 & 9.3 & -0.05 & 0.063 \\ Size & .. & .. & ... & ... \\ Speed & ... & ... & ... & ... \\ \end{matrix}$$

how would find any relationships between the data using V and S (using PCA) and am i making any mistakes?

This is probably a better fit at sister site Cross Validated

That said: PCA is a change of basis. What you're looking at is the change of basis matrix. This looks like a homework problem and the answer to your homework is "this is how we go from standard representation to representation by principal components".

Maybe that's not enough.

(1) From an abstract point of view, a basis for a vector spaces of finite dimension $$n$$ is a set of $$n$$ vectors $$e_1, \cdots, e_n$$ such that any vector $$v$$ can be expressed as a linear combination

$$v = \alpha_1 e_1 + \alpha_2 e_2 + \cdots + \alpha_n e_n$$

where $$\alpha_1, \cdots$$ are scalars. Now, there are many such bases. The obvious one is the canonical basis where $$e_1 = (1,0,\cdots), e_2 = (0,1,\cdots)$$. This one is implicit: when you get that a vector is $$v=(80, 10,30,1)$$ what is really meant is

$$v = 80 \begin{bmatrix}1\\ 0\\ 0 \\ 0\end{bmatrix} + 10 \begin{bmatrix}0\\ 1\\ 0 \\ 0\end{bmatrix} + 3 \begin{bmatrix}0\\ 0\\ 1 \\ 0\end{bmatrix} + 1 \begin{bmatrix}0\\ 0\\ 0 \\ 1\end{bmatrix}$$

Note that we're representing cars as vectors now. This means that expressions such as $$\text{Tesla} = \tfrac12 \text{Mercedes} + \tfrac12 \text{Prius}$$ (closure under sum) and $$\text{price} \cdot 2\text{Tesla} = 2\cdot \text{price } \text{Tesla}$$ (where $$\text{price}$$ is a scalar) make sense.

Which is ok, except that our dimensions are something like $$\text{noise, size, speed, mpg}$$ and the cars

$$e_1 = [\text{noise} = 1, \text{size} = 0, \text{speed} = 0, \text{mpg} = 0]$$

$$e_2 = [\text{noise} = 0, \text{size} = 1, \text{speed} = 0, \text{mpg} = 0]$$

$$e_3 = [\text{noise} = 0, \text{size} = 0, \text{speed} = 1, \text{mpg} = 0]$$

$$e_4 = [\text{noise} = 0, \text{size} = 0, \text{speed} = 0, \text{mpg} = 1]$$

are not terribly meaningful. Of course, sometimes something like regression analysis returns something valid, which means you have a scalar multiplier that relates, say, "1 size unit" to "contribution to price in USD".

Anyway, although $$e_1, e_2, e_3, e_4$$ have human nicknames, we might need better ways to specify a car.

(2) Pick, for example, four cars that are so completely different they should be considered unrelated and equally important. In linear algebra and statistics "unrelated" is represented by "orthogonal", while "equally important" means they have the same length (so taking an average of a Tesla and an El Camino works out correctly). An orthogonal, unit-length basis for the space of cars based on those four cars can be obtained via the Gram-Schmidt process.

Another way of obtaining orthogonal basis vectors is eigendecomposition. Square full rank matrices $$M$$ with $$n$$ rows/columns should have $$n$$ vectors $$v_1, \cdots$$ such that

$$M v_i = \lambda_i v_i$$

where $$\lambda_i$$ is a scalar. In other words, the matrix (which is given by your four cars -- since we have four dimensions) acts on these vectors (the eigenvectors) as if it was a scalar coefficient (the corresponding eigenvalue). And every other vector can be given as a combination of the eigenvectors, since this is a basis. In matrix form,

$$M = P^\top D P$$

where $$D$$ is a diagonal matrix with the eigenvalues and $$P$$ has the eigenvectors. Now, unlike the Gram-Schmidt basis, these basis vectors are not unit length; rather, they're associated with a scale factor. But if, say, the eigenvalue associated with $$v_4$$ is very small, this means the matrix affects $$v_4$$ very little. $$v_4$$ is likely to represent a component that's not too important. So you might as well lop off $$v_4$$; this will give you a basis $$v_1, v_2, v_3$$ that now spans a 3-dimensional space.

This can be really useful in situations where you have hundreds or thousands of column variables, most irrelevant or largely redundant.

(3) This has limitations for analyzing data matrices with $$m$$ observations as rows and $$n< column attributes. I've been telling you so far to choose the $$n$$ prototype observations that should be essential -- but your clients and investors may disagree.

We've been discussing matrices as "acting on" vectors, which means that they can be interpreted as linear maps. A rectangular matrix of data is also a linear map -- you can literally multiply your data $$M$$ by a vector of regression coefficients and obtain a vector of predictions, but it has "column spaces" and "row spaces" of different dimension. The "column space" is the space of vectors the matrix acts on. The "row space" is the space of vectors that left-multiply the matrix. (This is a little confusing conceptually but whenever lost remember the row space of $$A$$ is the column space of $$A^\top$$)

So for a data matrix, the column space is the space of features (car length, weight, etc.); it has as dimension the number of column variables you have. The row space is the space of observations. Think of it like this: the data, when transposed, is a transformation from a space of cars to a space of features.

The basic idea for SVD is to "eigendecompose in both directions", i.e. to find scalar singular values $$\sigma_1, \cdots$$ such that $$Ma = \sigma_i a_i$$

$$M^\top b = \sigma_i b_i$$

where $$a_i$$, $$b_i$$ are now basis vectors for the columnspace and rowspace respectively, i.e. observations for one column or columns for one observation. Again, if, say, $$\sigma_4$$ is very small, it means that the effect of the fourth singular vector isn't very relevant and we can delete that column.

In matrix form, this means obtaining the following decomposition

$$M = U\Sigma V^\top$$

where $$U$$ has the basis for the columnspace and $$V$$ the basis for the rowspace.

Carrying out the calculations with the matrix expression for the eigendecomposition will show that $$U, V$$ are obtained from the eigenvectors for $$MM^\top, M^\top M$$ and the diagonal matrix $$\Sigma = D^2$$ (where $$D$$ comes from either eigendecomposition).

I hope this helps you build intuition. It sure jogged my memory.

• so how would i find out any correlations.. how would i interpret the VS' (PC) matrix – Coder Mar 12 at 19:55
• Wait, what is V and what is S in your notation? – user8948 Mar 12 at 20:52
• using [U,S,V] = svd(data,0), i get the decomposed version of the data. S is the diagonal singular values, and i am not 100% sure but i think V is the principle components – Coder Mar 12 at 20:53