Shrinking eigenvalues as two columns become collinear In this answer to the question of why OLS estimates become unstable in the presence of correlation between predictors, I try to offer a mathematical explanation via the eigendecomposition  of the inverse of a Gram matrix.
I appeal to a suspicion that if two columns of a full rank matrix are approximately collinear, then one eigenvalue of the corresponding Gram matrix should be close to $0$.  I realized I can't prove this and was wondering if someone could help me prove it or demonstrate that this intuition fails.
Put more concretely:  Let $X$ be a full rank matrix so that for at least one pair of columns $X_i, X_j$ $X_j \approx X_j$ (hence $X_i$ and $X_j$ are highly correlated.  Perhaps the correlation is $\rho$).  Let $A = X^TX$ be a Gram Matrix.  Can we relate the maginitude of the smallest eigenvalue of $A$ to the correlation observed between $X_i$ and $X_j$?
My intuition says that since $X_i$ and $X_j$ are correlated, then $<X_i, Z> \approx <X_j, Z>$ for some vector $Z$.  This implies that two columns in $A$ are also correlated, and I suspect an eigenvalue would be close to $0$ because of this correlation.
 A: Say that the columns that are near collinear are $X_a$ and $X_b$. To define this, let's say that $X_b = \rho*X_a+
\frac{\vec{\epsilon}}{\vert\vert X_{max}\vert\vert}$, for some vector with small length $\vec{\epsilon}$ and for the column with maximum length $X_{max}$.
Recall that if $\lambda$ is an eigenvector of $A$, $A\vec{v} = \lambda\vec{v}$ for some eigenvector $\vec{V}$ by definition. Then observe that the $n$th element of $A\vec{v}$ is $\sum_{i} ((X_n \cdot X_i)v_i)$, where $v_i$ is the $i$th element of $\vec{v}$.
Therefore, $(A\vec{v})_a = \lambda {v}_a$ and $(A\vec{v})_b = \lambda {v}_b$
$$(A\vec{v})_a = \lambda \vec{v}_a \implies \sum_{i} ((X_a \cdot X_i)v_i) = \lambda \vec{v}_a$$
$$(A{v})_b = \lambda {v}_b \implies \sum_{i} ((X_b \cdot X_i)v_i) = \lambda {v}_b \implies \rho\sum_{i} ((X_a \cdot X_i)v_i)+\frac{1}{\vert\vert X_{max}\vert\vert}\sum_{i} ((\vec{\epsilon}\cdot X_i)v_i) = \lambda {v}_b$$
Therefore:
$$\rho\lambda {v}_a+\frac{1}{\vert\vert X_{max}\vert\vert}\sum_{i} ((\vec{\epsilon}\cdot X_i)v_i) = \lambda {v}_b \implies \frac{1}{\vert\vert X_{max}\vert\vert}\sum_{i} ((\vec{\epsilon}\cdot X_i)v_i) = \lambda(v_b-\rho v_a)$$
Observe that $\vec{\epsilon} \cdot X_i \leq ||\epsilon||*||X_i|| \leq ||\epsilon||*||X_{max}||$
Therefore:
$$||\epsilon|| \sum_{i} v_i \geq \lambda(v_b-\rho v_a) \implies \lambda \leq ||\epsilon|| \frac{\sum_{i} \frac{v_i}{v_a}}{\frac{v_b}{v_a}-\rho}$$.
So, as the span of epsilon decreases to zero, the eigenvalue (which always must be nonnegative for the symmetric matrix $A$) will go to zero by the Squeeze Theorem.
