I am trying to answer the question phrased as follows
Give a detailed explanation of Principal Component Analysis. Your explanation should include explanations of the terms: geometric information; covariance matrix; orthogonal transformation; Spectral Theorem and describe how the technique can be used to reduce dimensionality while retaining much geometric information
My understanding of Principal Component Analysis is that it reduces a number of variables x1, x2... to a smaller set of principal components that store as much of the original information from the original variables in these newly created principal components.
For example if one were to reduce two attributes of a car, say speed and engine size into one principal component. These original components would be plotted on an xy plane and then brought together into a new line of best fit, putting these points through an orthogonal transformation that preserves the points original distance from one another.
The covariance matrix measures how variations in pairs of variables are linked to each other and its diagonal values are always equal to 0. So in this example it would store the variance of the cars speeds and engine size.
The covariance matrix is then used to calculate the relevant set of eigenvalues and eigenvectors.
Dimensionality can be reduced by then selecting the k largest eigenvectors as the new k principal components which represent as much of the variance as possible with as few variables. The more the dimensionality is reduced (i.e. the more principal components that are removed) the less the variance of the original variables (or geometric information) is captured in the final result.
My two questions are
- How does the spectral theorem relate to PCA.
- Have I provided a detailed enough explanation of what PCA does otherwise.
Any help would be greataly appreciated!