In principal component analysis (PCA), one can choose either the covariance matrix or the correlation matrix to find the components. These give different results because, I suspect, the eigenvectors between both matrices are not equal. (Mathematically) similar matrices have the same eigenvalues, but not necessarily the same eigenvectors. Several questions: (1) Why this difference? (2) Does PCA make sense, if you can get two different answers? (3) Which of the two methods is 'best'? (4) Since PCA operates on standardized (not) raw data in both cases, i.e., scaled by their standard deviation, does it make sense to use the results to draw conclusions about the dominance of variation for the actual, unstandardized data?

  • $\begingroup$ If you scale them by their standard deviation, doesn't that make the covariance matrix into a correlation matrix? $\endgroup$ – Michael Hardy Jun 26 '13 at 13:09
  • $\begingroup$ This is more of a statistics question so is better asked at Cross Validated. You will probably get more/better answers there. $\endgroup$ – kjetil b halvorsen Jul 3 '14 at 9:14
  • $\begingroup$ See stats.stackexchange.com/questions/53/… and the questions linked from it $\endgroup$ – Henry May 1 '20 at 8:28

The problem with not standardizing, i.e. with not scaling the variables by their standard deviation, is that if, for example, one variable is measured in centimeters and another in dollars, then changing centimeters to meters can actually change the eigenvectors, so an arbitrary choice of units can alter the results. Hence I'd use the correlation matrix.

  • $\begingroup$ Correction to my part (4): "both cases" is incorrect; standardized variables are used in correlation-based PCA, not in covariance-based. But the issue and question still stands for the former. $\endgroup$ – Lucozade Jun 26 '13 at 13:23
  • $\begingroup$ Thanks Michael. Yes, this is the message/advice I am getting from literature too, but in case where the data are physically dimensionless, you still have a choice of two. It is not clear which one should be chosen on a more positive, fundamental basis. $\endgroup$ – Lucozade Jun 26 '13 at 13:29
  • $\begingroup$ My issue with scaling is that it seems to destroy the problem you are trying to solve. If you standardize each variable X by its own (= across different observations for the same variable) standard deviation, before performing correlation based PCA, how can it still make sense to look for directions of maximum variance for combinations of the variables, which is what PCA is all about? I know that that correlation based PCA is very convenient (standardized variables are dimensionless, so their linear combinations can be added; other advantages are also based on pragmatism), but is it correct? $\endgroup$ – Lucozade Jun 26 '13 at 23:04
  • $\begingroup$ It seems to me that covariance based PCA is the only truly correct one (even when the variances of the variables differ greatly) and that, whenever this version cannot be used, correlation based PCA should not be used either. $\endgroup$ – Lucozade Jun 26 '13 at 23:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.