8
$\begingroup$

Let matrix $A \in \mathbb{R}^{n\times n}$ be positive semidefinite.

  • Is it then true to that $$ (A + \lambda I)^{-1} \to \mathbf{0} \quad (\lambda \to \infty) \quad ? $$

  • If so, is the fact that $A$ is positive definite irrelevant here?


My thoughts so far: $$ (A + \lambda I)^{-1} = \Big(\lambda( \frac{1}{\lambda}A + I ) \Big)^{-1} = \frac{1}{\lambda} \Big(\frac{1}{\lambda}A + I \Big)^{-1} $$ I think that $\lim_{\lambda \to \infty} \Big( \frac{1}{\lambda}A + I \Big)^{-1} = I^{-1} = I$, but I don't know if I can just pass the $\lim$ through the inverse $(\cdot)^{-1}$ like that. If this is the case, then $$ \lim_{\lambda \to \infty} (A + \lambda I)^{-1} = \lim_{\lambda \to \infty} (1/\lambda) \lim_{\lambda \to \infty} (A/\lambda + I)^{-1} = 0 \cdot I = \mathbf{0} $$ as I'd like to show.


Where this comes from:

I'm trying to justify a claim made in an econometrics lecture. Namely,

$$ \textrm{Var}(\hat{\beta}^{\textrm{ridge}}) = \sigma^2 (X^{T}X + \lambda I)^{-1} X^T X [(X^T X + \lambda I)^{-1}]^T \to \mathbf{0} $$ where $\hat{\beta}^\textrm{ridge}$ is the ridge estimator in a linear model, $X \in \mathbb{R}^{n \times p}$ is the design matrix, and the equality is known. The limit, however, wasn't justified.

$\endgroup$
8
  • 6
    $\begingroup$ $A$ can be any matrix above. The point is, the inverse of a matrix is a continuous function in a neighbourhood of the identity, therefore since $A - \lambda I$ is going to eventually be invertible, we may pass the limit inside the inverse by continuity, giving the desired result by the continuity of scalar multiplication. $\endgroup$ Jan 24, 2019 at 17:27
  • 4
    $\begingroup$ If $\|\cdot\|$ is a matrix norm, then the Neumann series guarantees that $A+\lambda I$ is invertible with $$(A+\lambda I)^{-1} = \sum_{n=0}^{\infty} \frac{(-1)^n}{\lambda^{n+1}}A^n, $$ which converges uniformly on the region $|\lambda| \geq \|A\|+\delta$ for any given $\delta > 0$. By the Weierstrass M-test, the limit as $\lambda\to\infty$ can be evaluated term-wise, proving the desired claim. $\endgroup$ Jan 24, 2019 at 17:34
  • $\begingroup$ @астонвіллаолофмэллбэрг Great! That completes my line of reasoning. For others looking on, here's why there is a neighborhood of $I$ in $M_n(\mathbb{R})$ in which $(\cdot)^{-1}$ is continuous: $(\cdot)^{-1} : GL_n(\mathbb{R}) \to GL_n(\mathbb{R})$ is continuous and $GL_n(\mathbb{R})$ is open in $M_n(\mathbb{R})$ (see: math.stackexchange.com/a/810675/369800). [To understand the proof just linked: determinant continuous (see: math.stackexchange.com/a/121834/369800) and adjoint continuous (see: math.stackexchange.com/a/2031642/369800)] $\endgroup$
    – zxmkn
    Jan 24, 2019 at 19:07
  • $\begingroup$ Recall that the inverse matrix is the adjugate matrix divided by the discriminant. Thus a "singularity" of the inversion only happens when the discriminant vanishes. $\endgroup$
    – Alexey
    Jan 26, 2019 at 20:45
  • 1
    $\begingroup$ @Mah I would like to think we can do so, but the argument is likely to be more convoluted. I think because $B^TB$ is positive definite, we can lower bound the smallest eigenvalue of $A+\lambda B^TB$ so that it goes to infinity with $\lambda$, then we can be done. $\endgroup$ Feb 22, 2021 at 4:09

2 Answers 2

4
$\begingroup$

The eigenvalues of $A+\lambda I$ are of the form $\lambda+\mu$, where $\mu$ is an eigenvalue of $A$ (necessarily real). Then, for $\lambda$ sufficiently large, the eigenvalues of $A+\lambda I$ are all $>1$.

Note that a matrix $S$ that diagonalizes $A$ also diagonalizes $A+\lambda I$, let $A=SDS^{-1}$, with $D$ diagonal.

Then $(A+\lambda I)^{-1}$ is diagonalizable with eigenvalues in $(0,1)$ and therefore $$ \lim_{\lambda\to\infty}(A+\lambda I)^{-1}= S\Bigl(\,\lim_{\lambda\to\infty}(D+\lambda I)^{-1}\Bigr)S^{-1}=0 $$ It is not necessary that $A$ is semipositive definite. Any symmetric matrix will do.

$\endgroup$
3
$\begingroup$

The answer I liked the best was left in the comments by астон вілла олоф мэллбэрг, since it shows that $A$ does not need any special structure. Here I'm pulling his answer down and including a bit more detail.


We have $$ (A + \lambda I)^{-1} = \Big(\lambda( \frac{1}{\lambda}A + I ) \Big)^{-1} = \frac{1}{\lambda} \Big(\frac{1}{\lambda}A + I \Big)^{-1}, $$ and we claim that $\Big(\frac{1}{\lambda}A + I \Big)^{-1} \to I^{-1} = I \quad (\lambda \to \infty)$. Therefore, $$ (A + \lambda I)^{-1} = \frac{1}{\lambda} \Big(\frac{1}{\lambda}A + I \Big)^{-1} \to 0 \cdot I = \mathbf{0} \quad (\lambda \to \infty), $$ which was the desired result.

We complete the proof by showing the claim. Since $GL_n(\mathbb{R})$ is open in $M_n(\mathbb{R})$, we find some $\epsilon > 0 $ such that the open ball $B(I, \epsilon) \subseteq GL_n(\mathbb{R})$. Hence, for sufficiently large $\lambda$, we know that $(A/\lambda + I) \in B(I, \epsilon) \subseteq GL_n(\mathbb{R})$. Also knowing that $(\cdot)^{-1} : GL_n \to GL_n$ is continuous, we have $$ \lim_{\lambda \to \infty}\Big(\frac{1}{\lambda}A + I \Big)^{-1} = \Big(\lim_{\lambda \to \infty} \frac{1}{\lambda}A + I \Big)^{-1}= (I)^{-1} = I, $$ which completes the proof.


To understand the linked proof of the continuity of $(\cdot)^{-1}$, see here for justification that the determinant operator is continuous and here for justification that the adjoint operator is continuous.

$\endgroup$
1
  • 1
    $\begingroup$ I may not be астон вілла олоф мэллбэрг anymore, but Teresa is very pleased that her idea was useful, even if it was two years ago! $\endgroup$ Feb 22, 2021 at 4:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.