0
$\begingroup$

I'm not sure if I asked this question correctly. But it occurred to me that in PCA (aka SVD), we treat the data matrix as if it is a linear transformation by talking about its 'Eigen vectors/values' but Eigen vectors and values are implicitly supposed to define the behavior of a matrix as linear transformation, since it describes what happens when you transform special vectors with the matrix.

So could you explain to me how/if it is indeed valid to think of an arbitrary data matrix (e.g. an arbitrary data frame) as a linear transformation? Specifically when you have no intention of using it that way, and just want to do PCA on it...

$\endgroup$
7
  • $\begingroup$ ??? Any matrix multiplication is a linear transformation! A(X+ Y)= AX+ AY and A(cX)= cAx. It is a little harder to show that any linear transformation (on finite dimensional space) can be represented by a matrix. $\endgroup$
    – user247327
    Sep 20, 2021 at 21:40
  • $\begingroup$ I'm not talking about matrix multiplication I'm talking about any data matrix. I don't understand what Eigen vectors are supposed to represent in a matrix which is not intended to be used as a linear transformation. $\endgroup$
    – profPlum
    Sep 20, 2021 at 23:46
  • $\begingroup$ Then I don't understand what you are doing with matrices. The whole point of matrices is how they are multiplied! $\endgroup$
    – user247327
    Sep 21, 2021 at 0:06
  • $\begingroup$ The rank $p$ approximation obtained from the SVD of a matrix $X$ is the best rank $p$ approximation to $X$ in the sense of the operator norm and the Hilbert-Schmidt norm. So the SVD is useful for dimensionality reduction. $\endgroup$
    – Mason
    Sep 21, 2021 at 2:53
  • $\begingroup$ Also if the columns of $X$ are centered, then $\frac{1}{n - 1}X^T X$ is the sample covariance matrix. The singular values of $X$ are the square roots of the eigenvalues of $X^T X$. $\endgroup$
    – Mason
    Sep 21, 2021 at 3:03

1 Answer 1

1
$\begingroup$

As commented by Mason: Up to a factor $\frac{1}{n-1}\,$ the square matrix $C=X^\top X$ is the sample covariance matrix (provided the columns are centered). Since $C$ is symmetric it can be diagonalized by an orthogonal matrix $S$: $$ D=S^\top CS\,. $$ The columns of $S$ are the eigenvectors of $C$ and the diagonal elements are its eigenvalues. Obviously, $$ S DS^\top=X^\top X\,. $$ Lets write $\sqrt{D}={\rm diag}(\sqrt{d_1},\dots,\sqrt{d_n})\,.$ If $\boldsymbol{x}$ is a random vector whose elements are independent, have mean zero and variance one then the covariance matrix of $$ \boldsymbol{y}=\sqrt{D}S^\top\boldsymbol{x} $$ is easily seen to be $C=X^\top X\,.$ In other words:

  • the functional interpretation of the eigenvectors $S$ of $C$ is how you must linearly combine independent RVs such that they have the same covariance as your data $X$.

The whole idea of PCA is to find a variable transformation that transforms the independent RVs $\boldsymbol{x}$ into linear combinations $\boldsymbol{y}$ that have the same covariance matrix as the original data.

In that context, the covariance matrix $C$ is not interesting as a linear map. In PCA it is more interesting to ask how many elements of $\boldsymbol{x}$ are needed to explain most of the variance of the data.

You should probably work on a realistic numerical example to get your head around it.

$\endgroup$
2
  • $\begingroup$ Thanks, so it sounds like it is unrelated to Eigen vector's property that they scale only by a scaler when transformed by the matrix? $\endgroup$
    – profPlum
    Sep 22, 2021 at 19:48
  • $\begingroup$ @profPlum : See my edit. $\endgroup$
    – Kurt G.
    Sep 23, 2021 at 10:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .