Eigenvectors are axes, eigenvalues are distances along these axes
Eigenvectors are the base of dimensional reduction techniques like PCA (principal component analysis), extremely useful in situations where we want to reduce the number of dimensions to a more practical one.
Concrete example: We want to find similar pictures in a large set and show the relationships in 2D (we don't know what are the similarity criteria, not either their number):
The result above is obtained using a non guided simple dimension reduction technique. Probably it isn't very useful, but it illustrates the creation of partial clusters, e.g. the direction the person is looking to, or the color of the skin, or whether the person opens the mouth, or who the person is. While there are plenty of criteria measured, we are able to use a 2D system to reveal their effects combined, this is actually the expected benefit. The algorithm here found two super abstract criteria which are eigenvectors, and returned the corresponding pair of eigenvalues for each picture, used as individual coordinates to arrange the set.
Face features as eigenvector: Eigenface
Using eigenvectors is a base technique in face recognition where we want to associate a name to a person picture. The eigenvectors in this case are eigenfaces.
Imagine we got black and white images of 47x62 pixels which can have some gray attribute, we actually have data with a value in 1348 dimensions:
Not all pixels are critical in a given image. We want to reduce the number of dimensions to the "useful" ones, without losing the main features of the images. Say we want to move from 1348 to 150 dimensions. This is the usual way of pre-processing images before doing some image classification, like face recognition, in order to decrease CPU workload.
The reduction is done by finding eigenvectors of the input images, these eigenvectors can be seen as basis images, from which the complete (actually nearly complete) images can be reconstructed.
Below are the 32 first eigenvectors, out of 150 which were computed by some PCA, in order of usefulness, that is they are dimensions along which original images have the highest variance. These images are somehow like the major harmonics of a sound (obtained using a Fourier transform):
Note the three first eigenvectors are luminosity related, the main variance in images not shot in controlled conditions.
To reconstruct the images from these eigenvectors, we only need to know the associated eigenvalues. The information to store and/or process for each image is now a vector of 150 eigenvalues, instead of the original vector of 1348 pixel values. A large gain. Still not much information has been lost. Here is a subset of original images, and images reconstructed using the eigenvectors shown above:
The number of eigenvectors used (and their choice) determines how much variance from the pixels is lost. This number is labeled below as "number of components", from the name of the reduction technique used: The principal component analysis (PCA):
For this graph, we see:
- With the 32 eigenvectors shown above, we can reconstruct images and get back more than 80% of the original pictures.
- With 150 eigenvectors, more than 95% of the information can be retrieved.
This is somehow similar to image compression, but much more too, because the eigenvalues can be used directly to make a relation between the image and the name of the person, we just need to train the classifier with a set of known images associated with the correct category (here the name to associate).
(I took my examples from this excellent book on machine learning. Faces pictures are from the LFW dataset)