Why does the gradient descent direction end up being the direction of the eigenvector corresponding to the smallest eigenvalue of the Hessian I read that the direction of gradient descent when optimising a function $f(w)$ after several iterations will be the same as the direction of the eigenvector corresponding to the smallest eigenvalue of the Hessian $H$. How can we prove this is the case?
Edit: For this to apply there is an assumption that the smallest eigenvalue is sufficiently smaller than the second smallest.
 A: Assume that we have a quadratic function $f(\mathbf{w})=\frac{1}{2}\mathbf{w^T H w}-\mathbf{b^T w}+c$.  In order to minimize $f(\mathbf{x})$, the gradient descent updates the iteration step as following:
$$\mathbf{w}(k+1)=\mathbf{w}(k)-\alpha \nabla f$$ 
in which $\nabla f(\mathbf{w})=\mathbf{Hw}-\mathbf{b}$ if $\mathbf{H}$ is Hermitian. So our iteration step will now be 
$$\mathbf{w}(k+1)=\mathbf{w}(k)+\alpha (\mathbf{b}-\mathbf{Hw}(k))$$
Next let $\mathbf{w_o}$ be the optimal $\mathbf{w}$ that gives the minimum of $f$ and we define a coefficient error vector $\mathbf{r}(k)$ that represents the error between the $\mathbf{w}(k)$ and the optimum $\mathbf{w_o}$:
$$\mathbf{r}(k)=\mathbf{w}(k)-\mathbf{w_0}$$
Our iteration step can now alternatively be described as 
$$\begin{align} \mathbf{r}(k+1) &=\mathbf{r}(k)+\alpha (\mathbf{b-Hw}(k))\\
&=\mathbf{r}(k)+\alpha (\mathbf{Hw_o-Hw}(k)) \\
&=\mathbf{r}(k)-\alpha (\mathbf{Hr}(k)) \\
&=\left( \mathbf{I}-\alpha \mathbf{H} \right) \mathbf{r}(k) \\
&= \left( \mathbf{I}-\alpha \mathbf{H} \right)^{k+1} \mathbf{r}(0) \end{align}$$
Since $\mathbf{H}$ is Hermitian, we know that it is unitarily diagonalizable, i.e. $\mathbf{Q^T HQ}=\mathbf{\Sigma}$ in which $\mathbf{Q}$ is an unitary matrix. Multiply the above iteration step with $\mathbf{Q^T}$ i.e. projecting the coefficient error vector $\mathbf{r}(k)$ into the eigenspace of $\mathbf{H}$, we get
$$\begin{align} \mathbf{Q^T r}(k+1) &=\left(\mathbf{I}-\alpha \mathbf{Q^T H Q}\right) \mathbf{Q^T r}(k) \\
&= \pmatrix{(1-\alpha \lambda_0)^{k+1} & & & \\ 
& (1-\alpha \lambda_1)^{k+1} & & \\
 & & \ddots & \\
 & & & (1-\alpha \lambda_n)^{k+1} } \mathbf{Q^T r}(0) \end{align}$$ 
such that $\lambda_0 > \lambda_1 > \dots >\lambda_n$. Define $\mathbf{v}(k)=\mathbf{Q^T r}(k)$. This means that the ith element $v_i$ of the vector $\mathbf{v}(k)$ is the length of the vector $\mathbf{r}(k)$ along the ith eigenvector of the matrix $\mathbf{H}$. We have now:
$$\mathbf{v}(k+1)= \pmatrix{(1-\alpha \lambda_0)^{k+1} & & & \\ 
& (1-\alpha \lambda_1)^{k+1} & & \\
 & & \ddots & \\
 & & & (1-\alpha \lambda_n)^{k+1} } \mathbf{v}(0)$$
 In order for the gradient descent to converge to the true minimum $\mathbf{w_o}$ we will need $\mathbf{v}(k+1)$ to converge to $\mathbf{0}$ as k increases. This has set a limitation for the value of $\alpha$ we can choose. We need to have $0<\alpha < \frac{1}{\lambda_n}$ in order for $\vert (1-\alpha \lambda_i)\vert < 1$ to ensure convergence.  
And from the inequalities $\lambda_0>\dots >\lambda_n$, we know that $(1-\alpha \lambda_0)>\dots > (1-\alpha \lambda_n)$, which means that the element $v_i$ of the vector $\mathbf{v}$ will diminish quicker than the element $v_{i+1}$. So over time, we will have $v_n$ having the most significant value compared to the rest of $v_i$'s . When project to the eigenspace, the error vector $\mathbf{r}$ will now have more weights on the eigenvector that correpsonds to the smallest eigenvalue, aligning itself to that eigenvector.
