20
$\begingroup$

I know that this is meant to explain variance but the description on Wikipedia stinks and it is not clear how you can explain variance using this technique

Can anyone explain it in a simple way?

$\endgroup$
  • 6
    $\begingroup$ When you refer to a Wiki article, you should always link to it $\endgroup$ – Casebash Jul 29 '10 at 13:15
  • 1
    $\begingroup$ This question might be better asked on the stats site. Although not an exact dupe, the answers to "What are principal component scores" may help. $\endgroup$ – walkytalky Jul 29 '10 at 17:16
  • 1
    $\begingroup$ The Wikipedia article is admittedly not great; but other Google results might be better, like ordination.okstate.edu/PCA.htm . In any case, it might be a good idea as walkytalky says to ask this question on the stats site. $\endgroup$ – Qiaochu Yuan Jul 29 '10 at 17:53
  • 1
    $\begingroup$ PCA itself is a mathematical technique, primarily used by statisticians, so while I agree that more statisticians would probably be able to answer this question, I think a description/explanation of PCA falls under mathematics, and it is not off-topic here. $\endgroup$ – Larry Wang Jul 29 '10 at 23:44
  • 1
    $\begingroup$ This question got some good answers on the stats site at stats.stackexchange.com/q/2691/919 . They supplement the nice one provided by @Kaestur Hakarl here. $\endgroup$ – whuber Jan 6 '11 at 19:38
19
$\begingroup$

Principal component analysis is a useful technique when dealing with large datasets. In some fields, (bioinformatics, internet marketing, etc) we end up collecting data which has many thousands or tens of thousands of dimensions. Manipulating the data in this form is not desirable, because of practical considerations like memory and CPU time. However, we can't just arbitrarily ignore dimensions either. We might lose some of the information we are trying to capture!

Principal component analysis is a common method used to manage this tradeoff. The idea is that we can somehow select the 'most important' directions, and keep those, while throwing away the ones that contribute mostly noise.

For example, this picture shows a 2D dataset being mapped to one dimension: alt text
Note that the dimension chosen was not one of the original two: in general, it won't be, because that would mean your variables were uncorrelated to begin with.
We can also see that the direction of the principal component is the one that maximizes the variance of the projected data. This is what we mean by 'keeping as much information as possible.'

$\endgroup$
  • $\begingroup$ Nice succinct answer $\endgroup$ – Adrian Torrie Sep 22 '16 at 0:04
8
$\begingroup$

Spent the day learning PCA, hope my cartoon translates the intuition over to you!

I have also tried to briefly explain the utility of PCA and related it to an analogy (no maths) to help give that feeling of "learning closure".

Visual Intuition (zoom in)

enter image description here

Intuition via Utility

I think the main usage for PCA is to be able to categorise different distinct "things" e.g. Shiny cells vs. Dark cells in a way that leads to least error (in terms of predicting the right colour cell). E.g. Imagine sam was hiding behind me and I pinched a cell off the left side of his body then asked you to guess the color of the cell, by looking at the winning photo, or even the winning line, you can make a very good guess it will be a "dark cell".

Intuition via Analogy

So my understanding is that PCA is like taking a "picture" in a lower dimension, but the various methods used out there attempt to make the picture as informative as possible by deciding which "angle" to take the picture from (notice for 1D the angle of "squishing line" also vary).

Good video

http://www.youtube.com/watch?v=UUxIXU_Ob6E

$\endgroup$
1
$\begingroup$

PCA basically is a projection of a higher-dimensional space into a lower dimensional space while preserving as much information as possible.

I wrote a blog post where I explain PCA via the projection of a 3D-teapot...

enter image description here

...onto a 2D-plane while preserving as much information as possible:

enter image description here

Details and full R-code can be found in the post:
http://blog.ephorie.de/intuition-for-principal-component-analysis-pca

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.