Am I Misunderstanding Principal Component Analysis (PCA)?

Principal Component Analysis (PCA) has routinely caused me to question my understanding of mathematics, particularly linear algebra. Once again, PCA is present and I would like to engage to the community to confirm my previous formal education and recent references to refresh my understanding. To put it more succinct, is my interpretation of PCA (below), accurate?

PCA, as a dimension reduction technique is employed to reduce a large data-set (i.e. tons of variables) in to a more coherent and smaller data-set, while maintaining most of the 'principal' information.

This is where I am unsure of my understanding: The outputs from PCA - that is, the principal components are then utilized for further analysis. But to what end?

Let us take a routine example, the Iris data-set. Many programming (R, Python, SPS), focus on this data-set as a practical application of PCA. Note the output from Python's scikit-learn module:

Understanding that "identifies the combination of attributes..that account for the most variance in the data", what can we do with this output? My interpretation - which I belive is flawed is that - in the case of the PCA output (sckit-learn), is that there is a strong correlation between 'virginica' and 'versicolor'. But is that all? Is this merely a processing technique which is then fed to machine learning models? It does not seem that the outputs from PCA (e.g. PC1, PC2, PC3) could be used for feature reduction. Plotting principal components, what information are we getting - since we are not really outputting what the principal components hold? If one were to 'present' the output in 2-d (as with the 'PCA of IRIS dataset', below) - would the intent to be to show that there is a stronger correlation between 'virginica' and 'versicolor'?

I have interpreted PCA as a pre-processing technique that allows one to identify the 'most important' (weighted, influential, etc.) features. Thus,the output I would expect from PCA would by something more akin to:

• PC1: Petal Width
• PC2: Petal Length
• The first thing that I notice in that plot is that setosa is linearly separable from versicolor and virginica in that projection, so those features would be good to classify setosa vs not setosa, and still pretty good at distinguishing versicolor from virginica, although with some overlap. A class I took used PCA for biplots for exploratory analysis. I’ve used PCA mostly for dimension reduction, on datasets with thousands of features, bringing it down to 50-100 PC that account for 90-95% of the variation. But the PC won’t usually be features from the data like “petal length”.
– Joe
Apr 5 '21 at 10:42

Take a point cloud and/or a set of vectors (which are the same thing). Move the cloud to have $$0$$ mean. For every possible direction, determine the variance of the dataset along that direction. Declare the direction of maximal variance to be the first principal component.

Now, project out that component. The residual data is still mean zero, so compute the new direction of maximal variance and that's the second principal component.

Repeat until all variance has been projected out (or the residual variance is some tiny fraction of the initial variance).

The example shows a common pattern: the dispersion in the direction of the first PC is from $$-4$$ to $$4$$. The dispersion in the second PC is from $$-1.5$$ to $$1.5$$, already a factor of $$3$$ reduction. If this pattern continues, the the dispersion in the direction of every two more PCs will fall by about a factor of $$10$$. Many real datasets only have a small number of PCs before the residuals are very clearly noise.

So, what is the first principal component? It's the direction of maximal variance. What does that have to do with input features like petal width and petal length? Very little, although if these features lead to large dispersion, you should expect the PCAs to incorporate them. If relevant features appear with low correlation, they will largely land in different components. If they appear with high correlation, they will largely land in one component. Thus, PCs make some progress working with features that are not independent. (Since many real world datasets have lots of implicit correlations, only a few PCs are needed to extract most of the dispersion.)

This is linear algebra, so PCs are weighted sums of features. The first PC is a weighted sum of features that points along the direction of maximal variance. You will frequently read "that explains the most variance", but that use of "explains" asserts more causation than is actually present. The second PC is a weighted sum of features pointing along the direction of maximal variance after removing the variance "explained" by the first PC. And so on.

If you are lucky, the first few PCs are nearly parallel to feature axes, so each can be easily described as "capturing this feature" and "capturing that feature". But many datasets have implicit, unexpected, or unrecognized correlations, that lead to components mixing features. For instance, I would expect petal dimensions to be correlated, so I expect those features to appear mixed into one component.

The dataset you graph suggests that the first principal component is sufficient to discriminate setosa from the other categories, so can be discriminated by a very simple linear classifier. The other two categories are not strongly separated by the first two components. However, a new sample having a very extreme second coordinate in the PCs basis might be assignable to one of the versicolor (near $$(-1,1.5)$$) or virginica (near $$(4, -1.5)$$) categories.

• Thanks for the quick reply Eric. Regarding your notes on PC1 (discriminate analysis), as well as other notes by respondents - it seems that PCA is used to reduce dimensions for further processing by ML algorithms, is that correct? I am still having trouble grasping what information one could discern from visualizing the PCs. As you mentioned, data undergoing PCA will likely become 'mixed', so in projecting the data to 2-d (or even 3-d), we will not be able to refer to the original data and say 'Ah, so petal-length' and 'petal-width' are the most important features, I should examine those Apr 5 '21 at 11:05
• @OctoCatKnows : PCA is used to reduce dimensionality. It was around long before ML. Believing that individual features should explain dispersion makes the error: believing features are independent. Maybe they are, maybe they aren't -- PCAs give some insight into which features are independent and explanatory and which are correlated. Have you never wondered whether two features were giving you as much as two coordinates-worth of discrimination or were really only giving one (or somewhere in between)? Human height and weight are easily measured, but likely correlated in most datasets. Apr 5 '21 at 11:12
• but since PCA is outputs a 'new' feature - how does one know which features from the original data are 'principal'. For example, sepal-length may or may not be a better indicator of class but from PCA, how do we interpret that? I see an output and the associated plot - but all I can interpret from that is PC1 contains ~70% of the variability, but still lost on where the original features fit. Outside of ML, wouldn't we want to know which of the original features are contained within the PCs? Apr 5 '21 at 11:21
• @OctoCatKnows : Featers are not (usually) principal components. PCs are linear combinations of features. They're telling you how your coordinates should have been oriented to condense dispersion in the earliest components. Either your features are parallel to these new axes or they aren't -- PCA doesn't care. Your PCs are weighted combinations of features. I'll say it a third time: highly correlated features land in the same component because the data does not justify treating them as independent. Apr 5 '21 at 11:25
• I think that last part turned-on the bulb, I hope. With regard to Iris, if - for example, petal length and sepal length 'land' in the same component, then there is not a justification for 'keeping' both? But how do we know which features 'land' within the PC? I am thinking Eigenvectors...? Apr 5 '21 at 11:31

The output from PCA are linear combinations of the input columns, where the first principal component retains the most 'information', the second column the second most 'information' and so on.

If you have 1000 columns, you can run PCA on them and might discover that you can use e.g. the first 20 principal components and still retain 99% of the variance of the data. So then you use the principal components in some machine learning model, but now with only 20 columns.

It might be enlightening to consider a super simplified example where you have two columns that have a correlation of 1. Then you can 'collect' that information in one principal component.

Don't know if you are looking for a mathematical description of this, but if you are just interested in the machine learning and practical applications, then you could also try: https://stats.stackexchange.com/

• Thanks for the reply. Regarding the PCA outputs. These are not linked directly to the original features, correct? For example, PC1 is not the same as Column1 (of original data). It seems then that rather than referring back to the original - PCA is actually creating new variables. Apr 5 '21 at 11:13
• That is correct. It is a linear combination of several variables/columns, so it doesn't have any easy interpretation. I would consider it an artificial/new column that still retains information about the original data. Apr 5 '21 at 11:15
• So, then, is it correct to presume that - from PCA results, one would take the output (eigenvectors) and note 'new' measures? Perhaps Iris is not the best example outside of ML application? For example, support.minitab.com/en-us/minitab/18/help-and-how-to/… seems to show a more usable (outside ML) application. Apr 5 '21 at 11:28
• Yes, that sounds about right. If you have two columns in your data $c_1$ and $c_2$ you 'project' them into one new PCA column $p$ which is a linear combination of them. So, $p = a_1c_1 + a_2c_2$. This new column is abstract and is not always open to interpretation. Then you use the new PCA column in your model, so you get something like $y = \alpha p + \beta$. Then you can go back to your original data: $$y = \alpha\left(a_1c_1 + a_2c_2\right) + \beta$$ (Somewhat simplified). So when you use PCA you are putting two models on top of each other. Apr 5 '21 at 12:34
• The iris dataset is fine. You can also make your own data with 5 rows and examine it. Might be instructive. Apr 5 '21 at 12:36