# What is the functional interpretation of the Eigen vectors in PCA?

I'm not sure if I asked this question correctly. But it occurred to me that in PCA (aka SVD), we treat the data matrix as if it is a linear transformation by talking about its 'Eigen vectors/values' but Eigen vectors and values are implicitly supposed to define the behavior of a matrix as linear transformation, since it describes what happens when you transform special vectors with the matrix.

So could you explain to me how/if it is indeed valid to think of an arbitrary data matrix (e.g. an arbitrary data frame) as a linear transformation? Specifically when you have no intention of using it that way, and just want to do PCA on it...

• ??? Any matrix multiplication is a linear transformation! A(X+ Y)= AX+ AY and A(cX)= cAx. It is a little harder to show that any linear transformation (on finite dimensional space) can be represented by a matrix. Sep 20, 2021 at 21:40
• I'm not talking about matrix multiplication I'm talking about any data matrix. I don't understand what Eigen vectors are supposed to represent in a matrix which is not intended to be used as a linear transformation. Sep 20, 2021 at 23:46
• Then I don't understand what you are doing with matrices. The whole point of matrices is how they are multiplied! Sep 21, 2021 at 0:06
• The rank $p$ approximation obtained from the SVD of a matrix $X$ is the best rank $p$ approximation to $X$ in the sense of the operator norm and the Hilbert-Schmidt norm. So the SVD is useful for dimensionality reduction. Sep 21, 2021 at 2:53
• Also if the columns of $X$ are centered, then $\frac{1}{n - 1}X^T X$ is the sample covariance matrix. The singular values of $X$ are the square roots of the eigenvalues of $X^T X$. Sep 21, 2021 at 3:03

As commented by Mason: Up to a factor $$\frac{1}{n-1}\,$$ the square matrix $$C=X^\top X$$ is the sample covariance matrix (provided the columns are centered). Since $$C$$ is symmetric it can be diagonalized by an orthogonal matrix $$S$$: $$D=S^\top CS\,.$$ The columns of $$S$$ are the eigenvectors of $$C$$ and the diagonal elements are its eigenvalues. Obviously, $$S DS^\top=X^\top X\,.$$ Lets write $$\sqrt{D}={\rm diag}(\sqrt{d_1},\dots,\sqrt{d_n})\,.$$ If $$\boldsymbol{x}$$ is a random vector whose elements are independent, have mean zero and variance one then the covariance matrix of $$\boldsymbol{y}=\sqrt{D}S^\top\boldsymbol{x}$$ is easily seen to be $$C=X^\top X\,.$$ In other words:
• the functional interpretation of the eigenvectors $$S$$ of $$C$$ is how you must linearly combine independent RVs such that they have the same covariance as your data $$X$$.
The whole idea of PCA is to find a variable transformation that transforms the independent RVs $$\boldsymbol{x}$$ into linear combinations $$\boldsymbol{y}$$ that have the same covariance matrix as the original data.
In that context, the covariance matrix $$C$$ is not interesting as a linear map. In PCA it is more interesting to ask how many elements of $$\boldsymbol{x}$$ are needed to explain most of the variance of the data.