Why are eigenvectors important for Deep Learning applications? I know it is quite of a trite question to ask about the importance of eigenvectors, but I do not understand how they can be relevant for Deep Learning and when we can use them.
Any reference to the literature as well is of course appreciated.
 A: As far as I know, eigenvectors are not so important in deep learning. Deep learning is using highly nonlinear transformations. That is why concepts from linear algebra like eigenvalues and eigenvectors do not play a major role in this field. In linear methods from unsupervised learning (e.g. factor analysis also known as Principal Component Analysis) and supervised learning (e.g. discriminant analysis) eigenvectors are used.
In factor analysis (Principal Component Analysis, PCA), we take the Covariance matrix and we try to represent our data set in a new coordinate system that is transformed. Imagine you have images (which are high dimensional) as inputs. The goal is to find a new coordinate system which captures a maximal amount of variance in the images. It turns out that the directions of this new coordinate system are exactly aligned with the eigenvectors of the covariance matrix $\boldsymbol{\Sigma}$. And the associated eigenvalue divided by the total number of eigenvalues will tell you what percentage of the whole variance is captured by the corresponding eigenvector. These eigenvectors are called eigenfaces if you calculate the eigenvectors of the covariance matrices of images.
Formally this is equal to maximizing the variance in the transformed system with coordinate direction $\boldsymbol{w}$ subject to the constraint $\boldsymbol{w}^T\boldsymbol{w}=1$ (normalized coordinate direction). We can write this using the method of Lagrange multipliers.
$$\mathcal{L}(\boldsymbol{w})=\boldsymbol{w}^T\boldsymbol{\Sigma}\boldsymbol{w}+\lambda\left[ 1-\boldsymbol{w}^T\boldsymbol{w}\right]$$
The first term is the variance in the transformed system and the second term is the Lagrange multiplier ($\lambda$ will turn out to be the eigenvalue) multiplied by the constraint. The gradient with respect to $\boldsymbol{w}$ is given by
$$\text{grad}_{\boldsymbol{w}}\mathcal{L}=2\boldsymbol{\Sigma}\boldsymbol{w}-2\lambda\boldsymbol{w}.$$
If you set this equal to zero you will see that this is the eigenvalue equation of the covariance matrix $\boldsymbol{\Sigma w}=\lambda \boldsymbol{w}$.
In the linear discriminant analysis, we have a similar problem. But now we have a multivariate input space with different groups. The goal is to determine a new coordinate system which allows us to separate the groups as good as possible. There are two matrices (within group $\boldsymbol{S}_\text{w}$ and between group $\boldsymbol{S}_\text{b}$) that describe the goodness of the separation in the new coordinate frame. If we use the Fisher's criterion it will turn out that the eigenvectors of $$\boldsymbol{S}^{-1}_\text{w}\boldsymbol{S}_\text{b}$$ are the directions of optimal discrimination for the groups.
To conclude there might be other fields in machine learning where eigenvalues and eigenvectors are important. But the core of deep learning relies on nonlinear transformations. Eigenvalues and eigenvectors are a core concept from linear algebra but not for the description of nonlinear transformations. The examples that I gave are both linear models, hence linear algebra and eigenvalues and eigenvectors might be playing an important role (as we know they actually do).
