Does $A^{-1} = A_0^{-1} + O_p(n^{-1})$ where $A = A_0 + O_p(n^{-1})$? Let $A$ be a random matrix that converges in probability to a constant matrix $A_0$ at the rate $O_p(n^{-1})$:
$$
A = A_0 + O_p(n^{-1}).
$$
I want to know if $A^{-1} = A_0^{-1} + O_p(n^{-1})$?
Let $I$ be the identity matrix. Here is my attempt at proving it, which I'm not sure is valid mainly because I don't know if we are allowed to define a series expansion in the form of $O_p$ variables.
$$
\begin{align}
A^{-1} &= (A_0 + O_p(n^{-1}))^{-1} \\
&= (A_0(I + A_0^{-1}O_p(n^{-1}))^{-1} \\
&= (A_0(I + O_p(n^{-1}))^{-1} \\
&= (I + O_p(n^{-1}))^{-1}A_0^{-1}.
\end{align}
$$
Now we perform a Neumann series expansion which gives
$$
\begin{align}
(I + O_p(n^{-1}))^{-1}
&= \sum_{k=0}^\infty (-1)^{k} (O_p(n^{-1}))^{k} \\
&= I - O_p(n^{-1}) \\
&= I + O_p(n^{-1}).
\end{align}
$$
I have assumed here that $(O_p(n^{-1}))^0$, is this true?
Then we substitute this into $(I + O_p(n^{-1}))^{-1}A_0^{-1}$ to get
$$
\begin{align}
A^{-1} &= (I + O_p(n^{-1}))A_0^{-1} \\
&= A_0^{-1} + O_p(n^{-1})A_0^{-1} \\
&= A_0^{-1} + O_p(n^{-1}).
\end{align}
$$
In the deterministic case, if we wanted to invert the matrix $(I+K)$ we need to have $||K|| < 1$ to ensure convergence. So does this mean that in the probabilistic case, to invert the matrix $(I + O_p(n^{-1}))$ we need to have $||O_p(n^{-1})|| < 1$? So the norm of an $O_p(n^{-1})$ term is required to be less than one..does this phrase even make sense?
It seems an $O_p(n^{-1})$ term will be increasingly likely to have a value less than $1$ as $n \to $\infty$, since the probability is becoming more concentrated around zero, but I don't think it can ever be fully guaranteed to be below one?
So does $A^{-1} = A_0^{-1} + O_p(n^{-1})$ really hold and if it does is my proof valid? If not, what is a valid proof?
 A: Set $A_n$ be a random matrix which converges to some constant matrix $A_0$ in probability at rate $\mathcal{O}(n^{-1})$, as $n\rightarrow \infty$. That is, for all $\epsilon>0$, there exists a constants $M, N>0$, with the property that
$$ \mathbb{P}\left( \left\|A_N - A_0\right\| > \frac{M}{n} \right)< \epsilon \qquad , n\geq N.
$$
Under the assumption that $A_n$ and $A_0$ are invertible, we write $B_n = A_n - A_0$ and observe that by the Neumann series expansion gives
\begin{equation} A^{-1}_n = A^{-1}_0 \left( I + B_n A^{-1}_0\right)^{-1} = A^{-1}_0\left( I + \sum_{k=1}^{\infty} \left(-B_n A^{-1}_0\right)^k \right),
\end{equation}
at least whenever $\left\|B_n A^{-1}_0\right\|< 1$, which may possibly depend on $n$ and points in underlying sample space. Anyway, to remedy this effect, we can simply use the conditional probability to write
$$ \mathbb{P}\left( \left\|A^{-1}_n - A^{-1}_0\right\| > \frac{M}{n} \right) \leq \mathbb{P}\left( \left\|A^{-1}_n - A^{-1}_0\right\| > \frac{M}{n} , \left\|A_n - A_0\right\|< \frac{M_1}{n} \right) + \mathbb{P}\left(\left\|A_n - A_0\right\|\geq \frac{M_1}{n} \right),
$$
Where $n > M_1 ,M$, for appropriately chosen $M,M_1>0$ to be determined at the end.
In order to treat the first term, we primarily notice that the previously derived Neumann series expansion implies
$$ \left\|A^{-1}_n - A^{-1}_0\right\| \leq \frac{\left\|A^{-1}_0\right\|\cdot \left\|B_n A^{-1}_0\right\|}{1-\left\|B_n A^{-1}_0\right\|}, 
$$
whenever $\left\|B_n A^{-1}_0\right\| < 1$, which in particular holds as long as
$\left\|A_n - A_0\right\| < \frac{M_{1}}{n}$, for sufficiently large $n>1$. With this at hand we get
$$ \mathbb{P}\left( \left\|A^{-1}_n - A^{-1}_0\right\| > \frac{M}{n} , \left\|A_n - A_0\right\|< \frac{M_1}{n} \right) \leq \mathbb{P} \left(\frac{\left\|A^{-1}_0\right\|\left\|B_n A^{-1}_0\right\|}{1-\left\|B_n A^{-1}_0\right\|} > \frac{M}{n} \right)= \mathbb{P} \left( \left\|B_n A^{-1}_0\right\|> \frac{M/n}{M/n+ \left\|A^{-1}_0\right\|}\right)
$$
$$ \leq \mathbb{P}\left(\left\|A_n- A_0\right\| > \frac{1}{n}\frac{M \left\| A^{-1}_0\right\|^{-1}}{1+ \left\|A^{-1}_0\right\|} \right).
$$
Adding up and choosing $M_1, M$ accordingly, answers your question in the affirmative.
A: Given a stochastic $A$ and fixed $A_0$ of size $d\times d$ such that $A=A_0+O_p(n^{-1})$, we can show $A^{-1}=A_0^{-1}+O_p(n^{-1})$ by noticing that matrix inverse is differentiable, so by local linearity rates are preserved.$\DeclareMathOperator\vect{vec}$
Consider the vector-valued vector-input function $f$ over flattened invertible matrices where $ f(\vect X)=\vect X^{-1}$. This vector-valued function is differentiable on the open set of invertible matrices, such that, refering to the indices $ij, kl$ from the unflattened matrices,
$$
\partial_{ij}f_{kl}(\vect X)=-(X^{-1})_{jk}(X^{-1})_{li}\,.
$$
For any convex combination $A_t$ of $A_0,A$, we have on the event $A_t$ is invertible,
$$\|Df(\vect A_t)\|\le \|Df (A_t)\|_F\le \|\vect Df (A_t)\|_1=\|\vect A_t^{-1}\|_1^2\le d^2\|A_t^{-1}\|_F^2\,.$$
Then since $\|A-A_t\|_F=O_p(n^{-1})$ by the assumption, as the other answer shows with Neumann series expansion, we have $A_t^{-1}$ exists whp and converges entrywise in probability to $A^{-1}$, so $\|A_t^{-1}\|_F=O_p(\|A^{-1}\|_F)=O_p(1)$.
Thus whp $\|Df(\vect A_t)\|$ is bounded by some constant $M$ and by the mean value inequality on $f$, $\|A_0^{-1}-A^{-1}\|_F\le M\|A_0-A\|_F$, which implies the conclusion.
